{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "colab": {
      "name": "DQN_Plain.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DBU7Id8EUl_S",
        "colab_type": "text"
      },
      "source": [
        "# Deep Q-Network"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ozKRJjuUl_X",
        "colab_type": "text"
      },
      "source": [
        "In 2015, Google DeepMind ([Link](https://deepmind.com/research/dqn/)) published a paper in Nature magazine that combines a deep convolution neural network with reinforcement learning for the first time in order to master a range of Atari 2600 games. They used only the raw pixels and score as the inputs. They were able to use the convolution layer to translate the pixels.  \n",
        "\n",
        "The very simple description is that they replaced the Q table in a Q-Learner with a neural network. This allowed them to take advantage of neural networks but still use reinforcement learning. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "btpip0ixUl_c",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3d259f14-cdd3-463f-d65c-bfbebcb48315"
      },
      "source": [
        "#Imports\n",
        "import gym\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from collections import deque\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "#from keras.models import Sequential\n",
        "#from keras.layers import Dense\n",
        "#from keras.optimizers import Adam\n",
        "import random\n",
        "\n",
        "#Create Gym\n",
        "from gym import wrappers\n",
        "envCartPole = gym.make('CartPole-v1')\n",
        "envCartPole.seed(50) #Set the seed to keep the environment consistent across runs"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[50]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ro-MZX_Ul_2",
        "colab_type": "text"
      },
      "source": [
        "**Experience Replay**  \n",
        "Definition: A mechanism inspired by biology that randomizes over the data removing the correlation in the observation sequence and smoothing over changes in the data distribution.  \n",
        "\n",
        "To perform an experience replay, the algorithm stores all of the agents experiences {$s_t,a_t,r_t,s_{t+1}$} at each time step in a data set. Normally in a q-learner, we would run the update rule on them. But, with experience replay we just store them.  \n",
        "\n",
        "Later during the training process these replays will be drawn uniformly from the memory queue and be ran through the update rule. There are 2 ways to handle this and I have coded both in the past. The first is to run them on every loop and the other is to run them after X amount of runs. In this code below, I run them each time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5zUIBLpaUl_5",
        "colab_type": "text"
      },
      "source": [
        "**Side Track: Vectorization**  \n",
        "I am going to take some time to talk about vectorization. If you are experienced with python you can skip this part. But, I went from running 100 episodes in mutliple minutes to being able to run 500 in less than 1.  \n",
        "\n",
        "The idea is that you execute the same task on ALL entries in an array at the same time.  \n",
        "\n",
        "Old way:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gvo4BXBSUl_7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "2ab9f89a-881c-417a-d791-b1e02825d658"
      },
      "source": [
        "tmp_array = []\n",
        "for i in range(100):\n",
        "    tmp_array.append(i)\n",
        "    \n",
        "#Add 10 to each element\n",
        "for i in range(100):\n",
        "    tmp_array[i] += 10\n",
        "print(tmp_array)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1_yM9W2oUmAJ",
        "colab_type": "text"
      },
      "source": [
        "Vectorized"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6oiOCCTUUmAo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "8114d46d-4ec5-4a87-af4c-56d4f9722976"
      },
      "source": [
        "tmp_array = []\n",
        "for i in range(100):\n",
        "    tmp_array.append(i)\n",
        "    \n",
        "#Add 10 to each element\n",
        "tmp_array = np.array(tmp_array) + 10\n",
        "print(tmp_array)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ 10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27\n",
            "  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45\n",
            "  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63\n",
            "  64  65  66  67  68  69  70  71  72  73  74  75  76  77  78  79  80  81\n",
            "  82  83  84  85  86  87  88  89  90  91  92  93  94  95  96  97  98  99\n",
            " 100 101 102 103 104 105 106 107 108 109]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lIxscDgXUmA1",
        "colab_type": "text"
      },
      "source": [
        "Most of these vectorizations will be calling a method versus just adding 10 but this is a simple solution. You will see a TREMENDOUS speed up avoiding the loops in python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-RAMOfUyUmA2",
        "colab_type": "text"
      },
      "source": [
        "**CartPole Example**  \n",
        "Again we will use the [CartPole](https://gym.openai.com/envs/CartPole-v1/) environment from OpenAI.  \n",
        "\n",
        "The actions are 0 to push the cart to the left and 1 to push the cart to the right.  \n",
        "\n",
        "The continuous state space is an X coordinate for location, the velocity of the cart, the angle of the pole, and the velocity at the tip of the pole. The X coordinate goes from -4.8 to +4.8, velocity is -Inf to +Inf, angle of the pole goes from -24 degrees to +24 degrees, tip velocity is -Inf to +Inf. With all of the possible combinations you can see why we can't create a Q table for each one.  \n",
        "\n",
        "To \"solve\" this puzzle you have to have an average reward of > 195 over 100 consecutive episodes. One thing to note, I am hard capping the rewards at 210 so this number can't average above that and it also could potentially drive the average down."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yM8KTrAEUmA4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Global Variables\n",
        "EPISODES = 500\n",
        "TRAIN_END = 0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "agAQlTehUmBE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Hyper Parameters\n",
        "def discount_rate(): #Gamma\n",
        "    return 0.95\n",
        "\n",
        "def learning_rate(): #Alpha\n",
        "    return 0.001\n",
        "\n",
        "def batch_size(): #Size of the batch used in the experience replay\n",
        "    return 24"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NvepwOe7UmBL",
        "colab_type": "text"
      },
      "source": [
        "**Deep Q-Network Class**  \n",
        "The following class is the deep Q-network that is built using the neural network code from Keras.  \n",
        "**init**:  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This creates the class and sets the local parameters.  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;I use a *deque* for the local memory to hold the experiences and a keras model for the NN.  \n",
        "\n",
        "**build_model(self)**:  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This builds the NN. I am using sequential model. Each of the layers are *Dense* despite the fact the document talks about using *Convolution*. But, they are only using that because they need to convert pixels and I already have numbers.  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;I am using an input layer(4), 24 neuron layer, 24 neuron layer, and an output layer(2).  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;For calculating the loss I am using mean squared error.  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;For an optimizer I am using [Adam](https://arxiv.org/abs/1412.6980v8). It is a variant of gradient descent and you can read the technical document at the link. If you want a slightly lighter explaining you can check out [Machine Learning Mastery](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/). You could also use SGD (Stochastic Gradient Descent) but Adam gives me better results and seems to be the standard in most examples.  \n",
        "\n",
        "**action(self,state)**:  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This generates the action.  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Explore: I am using the epsilon like previous lessons.  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Exploit: I use the NN to grab the 2 possible actions and then grab the argmax to find the better one  \n",
        "\n",
        "**test_action(self,state)**:  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This generates the action when I am testing. I want to 100% exploit  \n",
        "\n",
        "**store(self, state, action, reward, nstate, done)**:  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This places the observables in memory  \n",
        "\n",
        "**experience_replay(self, batch_size)**:  \n",
        "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This is where the training occurs. We grab the sample batches and then use the NN to predict the optimal action.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P3MYaBiXUmBM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class DeepQNetwork():\n",
        "    def __init__(self, states, actions, alpha, gamma, epsilon,epsilon_min, epsilon_decay):\n",
        "        self.nS = states\n",
        "        self.nA = actions\n",
        "        self.memory = deque([], maxlen=2500)\n",
        "        self.alpha = alpha\n",
        "        self.gamma = gamma\n",
        "        #Explore/Exploit\n",
        "        self.epsilon = epsilon\n",
        "        self.epsilon_min = epsilon_min\n",
        "        self.epsilon_decay = epsilon_decay\n",
        "        self.model = self.build_model()\n",
        "        self.loss = []\n",
        "        \n",
        "    def build_model(self):\n",
        "        model = keras.Sequential() #linear stack of layers https://keras.io/models/sequential/\n",
        "        model.add(keras.layers.Dense(24, input_dim=self.nS, activation='relu')) #[Input] -> Layer 1\n",
        "        #   Dense: Densely connected layer https://keras.io/layers/core/\n",
        "        #   24: Number of neurons\n",
        "        #   input_dim: Number of input variables\n",
        "        #   activation: Rectified Linear Unit (relu) ranges >= 0\n",
        "        model.add(keras.layers.Dense(24, activation='relu')) #Layer 2 -> 3\n",
        "        model.add(keras.layers.Dense(self.nA, activation='linear')) #Layer 3 -> [output]\n",
        "        #   Size has to match the output (different actions)\n",
        "        #   Linear activation on the last layer\n",
        "        model.compile(loss='mean_squared_error', #Loss function: Mean Squared Error\n",
        "                      optimizer=keras.optimizers.Adam(lr=self.alpha)) #Optimaizer: Adam (Feel free to check other options)\n",
        "        return model\n",
        "\n",
        "    def action(self, state):\n",
        "        if np.random.rand() <= self.epsilon:\n",
        "            return random.randrange(self.nA) #Explore\n",
        "        action_vals = self.model.predict(state) #Exploit: Use the NN to predict the correct action from this state\n",
        "        return np.argmax(action_vals[0])\n",
        "\n",
        "    def test_action(self, state): #Exploit\n",
        "        action_vals = self.model.predict(state)\n",
        "        return np.argmax(action_vals[0])\n",
        "\n",
        "    def store(self, state, action, reward, nstate, done):\n",
        "        #Store the experience in memory\n",
        "        self.memory.append( (state, action, reward, nstate, done) )\n",
        "\n",
        "    def experience_replay(self, batch_size):\n",
        "        #Execute the experience replay\n",
        "        minibatch = random.sample( self.memory, batch_size ) #Randomly sample from memory\n",
        "\n",
        "        #Convert to numpy for speed by vectorization\n",
        "        x = []\n",
        "        y = []\n",
        "        np_array = np.array(minibatch)\n",
        "        st = np.zeros((0,self.nS)) #States\n",
        "        nst = np.zeros( (0,self.nS) )#Next States\n",
        "        for i in range(len(np_array)): #Creating the state and next state np arrays\n",
        "            st = np.append( st, np_array[i,0], axis=0)\n",
        "            nst = np.append( nst, np_array[i,3], axis=0)\n",
        "        st_predict = self.model.predict(st) #Here is the speedup! I can predict on the ENTIRE batch\n",
        "        nst_predict = self.model.predict(nst)\n",
        "        index = 0\n",
        "        for state, action, reward, nstate, done in minibatch:\n",
        "            x.append(state)\n",
        "            #Predict from state\n",
        "            nst_action_predict_model = nst_predict[index]\n",
        "            if done == True: #Terminal: Just assign reward much like {* (not done) - QB[state][action]}\n",
        "                target = reward\n",
        "            else:   #Non terminal\n",
        "                target = reward + self.gamma * np.amax(nst_action_predict_model)\n",
        "            target_f = st_predict[index]\n",
        "            target_f[action] = target\n",
        "            y.append(target_f)\n",
        "            index += 1\n",
        "        #Reshape for Keras Fit\n",
        "        x_reshape = np.array(x).reshape(batch_size,self.nS)\n",
        "        y_reshape = np.array(y)\n",
        "        epoch_count = 1 #Epochs is the number or iterations\n",
        "        hist = self.model.fit(x_reshape, y_reshape, epochs=epoch_count, verbose=0)\n",
        "        #Graph Losses\n",
        "        for i in range(epoch_count):\n",
        "            self.loss.append( hist.history['loss'][i] )\n",
        "        #Decay Epsilon\n",
        "        if self.epsilon > self.epsilon_min:\n",
        "            self.epsilon *= self.epsilon_decay"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kGOv_PhSUmBY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Create the agent\n",
        "nS = envCartPole.observation_space.shape[0] #This is only 4\n",
        "nA = envCartPole.action_space.n #Actions\n",
        "dqn = DeepQNetwork(nS, nA, learning_rate(), discount_rate(), 1, 0.001, 0.995 )\n",
        "\n",
        "batch_size = batch_size()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uT9QeK_QUmBf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "65163524-dd27-48a7-a896-f6ae9243fbb6"
      },
      "source": [
        "#Training\n",
        "rewards = [] #Store rewards for graphing\n",
        "epsilons = [] # Store the Explore/Exploit\n",
        "TEST_Episodes = 0\n",
        "for e in range(EPISODES):\n",
        "    state = envCartPole.reset()\n",
        "    state = np.reshape(state, [1, nS]) # Resize to store in memory to pass to .predict\n",
        "    tot_rewards = 0\n",
        "    for time in range(210): #200 is when you \"solve\" the game. This can continue forever as far as I know\n",
        "        action = dqn.action(state)\n",
        "        nstate, reward, done, _ = envCartPole.step(action)\n",
        "        nstate = np.reshape(nstate, [1, nS])\n",
        "        tot_rewards += reward\n",
        "        dqn.store(state, action, reward, nstate, done) # Resize to store in memory to pass to .predict\n",
        "        state = nstate\n",
        "        #done: CartPole fell. \n",
        "        #time == 209: CartPole stayed upright\n",
        "        if done or time == 209:\n",
        "            rewards.append(tot_rewards)\n",
        "            epsilons.append(dqn.epsilon)\n",
        "            print(\"episode: {}/{}, score: {}, e: {}\"\n",
        "                  .format(e, EPISODES, tot_rewards, dqn.epsilon))\n",
        "            break\n",
        "        #Experience Replay\n",
        "        if len(dqn.memory) > batch_size:\n",
        "            dqn.experience_replay(batch_size)\n",
        "    #If our current NN passes we are done\n",
        "    #I am going to use the last 5 runs\n",
        "    if len(rewards) > 5 and np.average(rewards[-5:]) > 195:\n",
        "        #Set the rest of the EPISODES for testing\n",
        "        TEST_Episodes = EPISODES - e\n",
        "        TRAIN_END = e\n",
        "        break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "episode: 0/500, score: 29.0, e: 0.9801495006250001\n",
            "episode: 1/500, score: 22.0, e: 0.8822202429488013\n",
            "episode: 2/500, score: 13.0, e: 0.8307187014821328\n",
            "episode: 3/500, score: 10.0, e: 0.7940753492934954\n",
            "episode: 4/500, score: 13.0, e: 0.7477194593032545\n",
            "episode: 5/500, score: 10.0, e: 0.7147372386831305\n",
            "episode: 6/500, score: 11.0, e: 0.6797938283326578\n",
            "episode: 7/500, score: 19.0, e: 0.6211445383053219\n",
            "episode: 8/500, score: 22.0, e: 0.5590843898207511\n",
            "episode: 9/500, score: 18.0, e: 0.5134164023722473\n",
            "episode: 10/500, score: 10.0, e: 0.4907693883854626\n",
            "episode: 11/500, score: 10.0, e: 0.46912134373457726\n",
            "episode: 12/500, score: 13.0, e: 0.4417353564707963\n",
            "episode: 13/500, score: 10.0, e: 0.4222502236424958\n",
            "episode: 14/500, score: 9.0, e: 0.40565285250151817\n",
            "episode: 15/500, score: 10.0, e: 0.3877593341372176\n",
            "episode: 16/500, score: 10.0, e: 0.3706551064126331\n",
            "episode: 17/500, score: 11.0, e: 0.35253382661792404\n",
            "episode: 18/500, score: 11.0, e: 0.3352984938281715\n",
            "episode: 19/500, score: 12.0, e: 0.3173112652388396\n",
            "episode: 20/500, score: 12.0, e: 0.30028896908517405\n",
            "episode: 21/500, score: 10.0, e: 0.28704309604425327\n",
            "episode: 22/500, score: 10.0, e: 0.2743815040481898\n",
            "episode: 23/500, score: 12.0, e: 0.25966219297210513\n",
            "episode: 24/500, score: 13.0, e: 0.24450384299593592\n",
            "episode: 25/500, score: 9.0, e: 0.23489314109365644\n",
            "episode: 26/500, score: 9.0, e: 0.22566020663225933\n",
            "episode: 27/500, score: 12.0, e: 0.21355457002808648\n",
            "episode: 28/500, score: 12.0, e: 0.20209834538617025\n",
            "episode: 29/500, score: 10.0, e: 0.19318370215794672\n",
            "episode: 30/500, score: 11.0, e: 0.18373897616330553\n",
            "episode: 31/500, score: 11.0, e: 0.17475600159032884\n",
            "episode: 32/500, score: 9.0, e: 0.1678868750508869\n",
            "episode: 33/500, score: 12.0, e: 0.15888051309497406\n",
            "episode: 34/500, score: 9.0, e: 0.1526354036900377\n",
            "episode: 35/500, score: 14.0, e: 0.14300635237083656\n",
            "episode: 36/500, score: 8.0, e: 0.13807558583895513\n",
            "episode: 37/500, score: 11.0, e: 0.1313250884614265\n",
            "episode: 38/500, score: 9.0, e: 0.1261630989318213\n",
            "episode: 39/500, score: 10.0, e: 0.12059799144222175\n",
            "episode: 40/500, score: 12.0, e: 0.11412846151764894\n",
            "episode: 41/500, score: 10.0, e: 0.10909420695870241\n",
            "episode: 42/500, score: 10.0, e: 0.1042820154910064\n",
            "episode: 43/500, score: 10.0, e: 0.0996820918179746\n",
            "episode: 44/500, score: 10.0, e: 0.09528507271768329\n",
            "episode: 45/500, score: 10.0, e: 0.09108200798387568\n",
            "episode: 46/500, score: 9.0, e: 0.08750185146499175\n",
            "episode: 47/500, score: 9.0, e: 0.08406242000238873\n",
            "episode: 48/500, score: 10.0, e: 0.08035439121179945\n",
            "episode: 49/500, score: 11.0, e: 0.07642587550895225\n",
            "episode: 50/500, score: 10.0, e: 0.07305469791585968\n",
            "episode: 51/500, score: 10.0, e: 0.06983222438783\n",
            "episode: 52/500, score: 10.0, e: 0.0667518955258533\n",
            "episode: 53/500, score: 20.0, e: 0.060687903789832374\n",
            "episode: 54/500, score: 16.0, e: 0.056292216338080694\n",
            "episode: 55/500, score: 14.0, e: 0.052741004581916356\n",
            "episode: 56/500, score: 12.0, e: 0.049911691230058335\n",
            "episode: 57/500, score: 15.0, e: 0.04652918187562211\n",
            "episode: 58/500, score: 15.0, e: 0.043375904776212296\n",
            "episode: 59/500, score: 15.0, e: 0.04043632488927963\n",
            "episode: 60/500, score: 11.0, e: 0.03845939824099909\n",
            "episode: 61/500, score: 15.0, e: 0.03585300941485119\n",
            "episode: 62/500, score: 19.0, e: 0.03275978694079333\n",
            "episode: 63/500, score: 17.0, e: 0.030235026982422884\n",
            "episode: 64/500, score: 17.0, e: 0.0279048474362789\n",
            "episode: 65/500, score: 14.0, e: 0.026144461565619025\n",
            "episode: 66/500, score: 15.0, e: 0.02437265452736848\n",
            "episode: 67/500, score: 20.0, e: 0.022158551474944856\n",
            "episode: 68/500, score: 19.0, e: 0.020246819920648168\n",
            "episode: 69/500, score: 15.0, e: 0.018874695352408037\n",
            "episode: 70/500, score: 16.0, e: 0.0175075817047932\n",
            "episode: 71/500, score: 16.0, e: 0.016239489508417658\n",
            "episode: 72/500, score: 11.0, e: 0.015445542985048652\n",
            "episode: 73/500, score: 13.0, e: 0.014543875539898159\n",
            "episode: 74/500, score: 18.0, e: 0.01335588042198471\n",
            "episode: 75/500, score: 20.0, e: 0.012142582314594924\n",
            "episode: 76/500, score: 16.0, e: 0.011263082556340478\n",
            "episode: 77/500, score: 17.0, e: 0.01039504944313793\n",
            "episode: 78/500, score: 15.0, e: 0.009690578183705511\n",
            "episode: 79/500, score: 15.0, e: 0.009033848857400105\n",
            "episode: 80/500, score: 17.0, e: 0.008337620279773651\n",
            "episode: 81/500, score: 32.0, e: 0.007137688903470108\n",
            "episode: 82/500, score: 18.0, e: 0.006554657266046809\n",
            "episode: 83/500, score: 12.0, e: 0.006203029922830858\n",
            "episode: 84/500, score: 20.0, e: 0.005639523495125475\n",
            "episode: 85/500, score: 20.0, e: 0.005127208097935121\n",
            "episode: 86/500, score: 21.0, e: 0.004638126179940997\n",
            "episode: 87/500, score: 21.0, e: 0.004195697551210702\n",
            "episode: 88/500, score: 25.0, e: 0.003720129962782708\n",
            "episode: 89/500, score: 17.0, e: 0.0034334237278812784\n",
            "episode: 90/500, score: 31.0, e: 0.0029540634996138766\n",
            "episode: 91/500, score: 27.0, e: 0.0025931039452239623\n",
            "episode: 92/500, score: 30.0, e: 0.002242277027600331\n",
            "episode: 93/500, score: 26.0, e: 0.0019781821840480804\n",
            "episode: 94/500, score: 15.0, e: 0.0018441210136605677\n",
            "episode: 95/500, score: 15.0, e: 0.0017191451527812486\n",
            "episode: 96/500, score: 27.0, e: 0.0015090813310791721\n",
            "episode: 97/500, score: 60.0, e: 0.0011227276295345597\n",
            "episode: 98/500, score: 25.0, e: 0.0009954703940636294\n",
            "episode: 99/500, score: 48.0, e: 0.0009954703940636294\n",
            "episode: 100/500, score: 49.0, e: 0.0009954703940636294\n",
            "episode: 101/500, score: 48.0, e: 0.0009954703940636294\n",
            "episode: 102/500, score: 107.0, e: 0.0009954703940636294\n",
            "episode: 103/500, score: 42.0, e: 0.0009954703940636294\n",
            "episode: 104/500, score: 81.0, e: 0.0009954703940636294\n",
            "episode: 105/500, score: 67.0, e: 0.0009954703940636294\n",
            "episode: 106/500, score: 41.0, e: 0.0009954703940636294\n",
            "episode: 107/500, score: 89.0, e: 0.0009954703940636294\n",
            "episode: 108/500, score: 51.0, e: 0.0009954703940636294\n",
            "episode: 109/500, score: 70.0, e: 0.0009954703940636294\n",
            "episode: 110/500, score: 24.0, e: 0.0009954703940636294\n",
            "episode: 111/500, score: 42.0, e: 0.0009954703940636294\n",
            "episode: 112/500, score: 37.0, e: 0.0009954703940636294\n",
            "episode: 113/500, score: 141.0, e: 0.0009954703940636294\n",
            "episode: 114/500, score: 77.0, e: 0.0009954703940636294\n",
            "episode: 115/500, score: 61.0, e: 0.0009954703940636294\n",
            "episode: 116/500, score: 47.0, e: 0.0009954703940636294\n",
            "episode: 117/500, score: 29.0, e: 0.0009954703940636294\n",
            "episode: 118/500, score: 82.0, e: 0.0009954703940636294\n",
            "episode: 119/500, score: 44.0, e: 0.0009954703940636294\n",
            "episode: 120/500, score: 131.0, e: 0.0009954703940636294\n",
            "episode: 121/500, score: 87.0, e: 0.0009954703940636294\n",
            "episode: 122/500, score: 41.0, e: 0.0009954703940636294\n",
            "episode: 123/500, score: 156.0, e: 0.0009954703940636294\n",
            "episode: 124/500, score: 138.0, e: 0.0009954703940636294\n",
            "episode: 125/500, score: 147.0, e: 0.0009954703940636294\n",
            "episode: 126/500, score: 106.0, e: 0.0009954703940636294\n",
            "episode: 127/500, score: 132.0, e: 0.0009954703940636294\n",
            "episode: 128/500, score: 92.0, e: 0.0009954703940636294\n",
            "episode: 129/500, score: 177.0, e: 0.0009954703940636294\n",
            "episode: 130/500, score: 122.0, e: 0.0009954703940636294\n",
            "episode: 131/500, score: 116.0, e: 0.0009954703940636294\n",
            "episode: 132/500, score: 170.0, e: 0.0009954703940636294\n",
            "episode: 133/500, score: 192.0, e: 0.0009954703940636294\n",
            "episode: 134/500, score: 188.0, e: 0.0009954703940636294\n",
            "episode: 135/500, score: 169.0, e: 0.0009954703940636294\n",
            "episode: 136/500, score: 169.0, e: 0.0009954703940636294\n",
            "episode: 137/500, score: 197.0, e: 0.0009954703940636294\n",
            "episode: 138/500, score: 210.0, e: 0.0009954703940636294\n",
            "episode: 139/500, score: 210.0, e: 0.0009954703940636294\n",
            "episode: 140/500, score: 210.0, e: 0.0009954703940636294\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wRiFK9DpUmBq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "4383c6f9-1007-48d9-e2fc-6addbaa3c0bd"
      },
      "source": [
        "#Test the agent that was trained\n",
        "#   In this section we ALWAYS use exploit don't train any more\n",
        "for e_test in range(TEST_Episodes):\n",
        "    state = envCartPole.reset()\n",
        "    state = np.reshape(state, [1, nS])\n",
        "    tot_rewards = 0\n",
        "    for t_test in range(210):\n",
        "        action = dqn.test_action(state)\n",
        "        nstate, reward, done, _ = envCartPole.step(action)\n",
        "        nstate = np.reshape( nstate, [1, nS])\n",
        "        tot_rewards += reward\n",
        "        #DON'T STORE ANYTHING DURING TESTING\n",
        "        state = nstate\n",
        "        #done: CartPole fell. \n",
        "        #t_test == 209: CartPole stayed upright\n",
        "        if done or t_test == 209: \n",
        "            rewards.append(tot_rewards)\n",
        "            epsilons.append(0) #We are doing full exploit\n",
        "            print(\"episode: {}/{}, score: {}, e: {}\"\n",
        "                  .format(e_test, TEST_Episodes, tot_rewards, 0))\n",
        "            break;"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "episode: 0/360, score: 210.0, e: 0\n",
            "episode: 1/360, score: 210.0, e: 0\n",
            "episode: 2/360, score: 210.0, e: 0\n",
            "episode: 3/360, score: 210.0, e: 0\n",
            "episode: 4/360, score: 210.0, e: 0\n",
            "episode: 5/360, score: 210.0, e: 0\n",
            "episode: 6/360, score: 210.0, e: 0\n",
            "episode: 7/360, score: 210.0, e: 0\n",
            "episode: 8/360, score: 210.0, e: 0\n",
            "episode: 9/360, score: 210.0, e: 0\n",
            "episode: 10/360, score: 210.0, e: 0\n",
            "episode: 11/360, score: 210.0, e: 0\n",
            "episode: 12/360, score: 210.0, e: 0\n",
            "episode: 13/360, score: 210.0, e: 0\n",
            "episode: 14/360, score: 210.0, e: 0\n",
            "episode: 15/360, score: 210.0, e: 0\n",
            "episode: 16/360, score: 210.0, e: 0\n",
            "episode: 17/360, score: 210.0, e: 0\n",
            "episode: 18/360, score: 210.0, e: 0\n",
            "episode: 19/360, score: 210.0, e: 0\n",
            "episode: 20/360, score: 210.0, e: 0\n",
            "episode: 21/360, score: 210.0, e: 0\n",
            "episode: 22/360, score: 210.0, e: 0\n",
            "episode: 23/360, score: 210.0, e: 0\n",
            "episode: 24/360, score: 210.0, e: 0\n",
            "episode: 25/360, score: 210.0, e: 0\n",
            "episode: 26/360, score: 210.0, e: 0\n",
            "episode: 27/360, score: 210.0, e: 0\n",
            "episode: 28/360, score: 210.0, e: 0\n",
            "episode: 29/360, score: 210.0, e: 0\n",
            "episode: 30/360, score: 210.0, e: 0\n",
            "episode: 31/360, score: 210.0, e: 0\n",
            "episode: 32/360, score: 210.0, e: 0\n",
            "episode: 33/360, score: 210.0, e: 0\n",
            "episode: 34/360, score: 210.0, e: 0\n",
            "episode: 35/360, score: 210.0, e: 0\n",
            "episode: 36/360, score: 210.0, e: 0\n",
            "episode: 37/360, score: 210.0, e: 0\n",
            "episode: 38/360, score: 210.0, e: 0\n",
            "episode: 39/360, score: 210.0, e: 0\n",
            "episode: 40/360, score: 210.0, e: 0\n",
            "episode: 41/360, score: 210.0, e: 0\n",
            "episode: 42/360, score: 210.0, e: 0\n",
            "episode: 43/360, score: 210.0, e: 0\n",
            "episode: 44/360, score: 210.0, e: 0\n",
            "episode: 45/360, score: 210.0, e: 0\n",
            "episode: 46/360, score: 210.0, e: 0\n",
            "episode: 47/360, score: 210.0, e: 0\n",
            "episode: 48/360, score: 210.0, e: 0\n",
            "episode: 49/360, score: 210.0, e: 0\n",
            "episode: 50/360, score: 210.0, e: 0\n",
            "episode: 51/360, score: 210.0, e: 0\n",
            "episode: 52/360, score: 210.0, e: 0\n",
            "episode: 53/360, score: 210.0, e: 0\n",
            "episode: 54/360, score: 210.0, e: 0\n",
            "episode: 55/360, score: 210.0, e: 0\n",
            "episode: 56/360, score: 210.0, e: 0\n",
            "episode: 57/360, score: 210.0, e: 0\n",
            "episode: 58/360, score: 210.0, e: 0\n",
            "episode: 59/360, score: 210.0, e: 0\n",
            "episode: 60/360, score: 210.0, e: 0\n",
            "episode: 61/360, score: 210.0, e: 0\n",
            "episode: 62/360, score: 210.0, e: 0\n",
            "episode: 63/360, score: 210.0, e: 0\n",
            "episode: 64/360, score: 210.0, e: 0\n",
            "episode: 65/360, score: 210.0, e: 0\n",
            "episode: 66/360, score: 210.0, e: 0\n",
            "episode: 67/360, score: 210.0, e: 0\n",
            "episode: 68/360, score: 210.0, e: 0\n",
            "episode: 69/360, score: 210.0, e: 0\n",
            "episode: 70/360, score: 210.0, e: 0\n",
            "episode: 71/360, score: 210.0, e: 0\n",
            "episode: 72/360, score: 210.0, e: 0\n",
            "episode: 73/360, score: 210.0, e: 0\n",
            "episode: 74/360, score: 210.0, e: 0\n",
            "episode: 75/360, score: 210.0, e: 0\n",
            "episode: 76/360, score: 210.0, e: 0\n",
            "episode: 77/360, score: 210.0, e: 0\n",
            "episode: 78/360, score: 210.0, e: 0\n",
            "episode: 79/360, score: 210.0, e: 0\n",
            "episode: 80/360, score: 210.0, e: 0\n",
            "episode: 81/360, score: 210.0, e: 0\n",
            "episode: 82/360, score: 210.0, e: 0\n",
            "episode: 83/360, score: 210.0, e: 0\n",
            "episode: 84/360, score: 210.0, e: 0\n",
            "episode: 85/360, score: 210.0, e: 0\n",
            "episode: 86/360, score: 210.0, e: 0\n",
            "episode: 87/360, score: 210.0, e: 0\n",
            "episode: 88/360, score: 210.0, e: 0\n",
            "episode: 89/360, score: 210.0, e: 0\n",
            "episode: 90/360, score: 210.0, e: 0\n",
            "episode: 91/360, score: 210.0, e: 0\n",
            "episode: 92/360, score: 210.0, e: 0\n",
            "episode: 93/360, score: 210.0, e: 0\n",
            "episode: 94/360, score: 210.0, e: 0\n",
            "episode: 95/360, score: 210.0, e: 0\n",
            "episode: 96/360, score: 210.0, e: 0\n",
            "episode: 97/360, score: 210.0, e: 0\n",
            "episode: 98/360, score: 210.0, e: 0\n",
            "episode: 99/360, score: 210.0, e: 0\n",
            "episode: 100/360, score: 210.0, e: 0\n",
            "episode: 101/360, score: 210.0, e: 0\n",
            "episode: 102/360, score: 210.0, e: 0\n",
            "episode: 103/360, score: 210.0, e: 0\n",
            "episode: 104/360, score: 210.0, e: 0\n",
            "episode: 105/360, score: 210.0, e: 0\n",
            "episode: 106/360, score: 210.0, e: 0\n",
            "episode: 107/360, score: 210.0, e: 0\n",
            "episode: 108/360, score: 210.0, e: 0\n",
            "episode: 109/360, score: 210.0, e: 0\n",
            "episode: 110/360, score: 210.0, e: 0\n",
            "episode: 111/360, score: 210.0, e: 0\n",
            "episode: 112/360, score: 210.0, e: 0\n",
            "episode: 113/360, score: 210.0, e: 0\n",
            "episode: 114/360, score: 210.0, e: 0\n",
            "episode: 115/360, score: 210.0, e: 0\n",
            "episode: 116/360, score: 210.0, e: 0\n",
            "episode: 117/360, score: 210.0, e: 0\n",
            "episode: 118/360, score: 210.0, e: 0\n",
            "episode: 119/360, score: 210.0, e: 0\n",
            "episode: 120/360, score: 210.0, e: 0\n",
            "episode: 121/360, score: 210.0, e: 0\n",
            "episode: 122/360, score: 210.0, e: 0\n",
            "episode: 123/360, score: 210.0, e: 0\n",
            "episode: 124/360, score: 210.0, e: 0\n",
            "episode: 125/360, score: 210.0, e: 0\n",
            "episode: 126/360, score: 210.0, e: 0\n",
            "episode: 127/360, score: 210.0, e: 0\n",
            "episode: 128/360, score: 210.0, e: 0\n",
            "episode: 129/360, score: 210.0, e: 0\n",
            "episode: 130/360, score: 210.0, e: 0\n",
            "episode: 131/360, score: 210.0, e: 0\n",
            "episode: 132/360, score: 210.0, e: 0\n",
            "episode: 133/360, score: 210.0, e: 0\n",
            "episode: 134/360, score: 210.0, e: 0\n",
            "episode: 135/360, score: 210.0, e: 0\n",
            "episode: 136/360, score: 210.0, e: 0\n",
            "episode: 137/360, score: 210.0, e: 0\n",
            "episode: 138/360, score: 210.0, e: 0\n",
            "episode: 139/360, score: 210.0, e: 0\n",
            "episode: 140/360, score: 210.0, e: 0\n",
            "episode: 141/360, score: 210.0, e: 0\n",
            "episode: 142/360, score: 210.0, e: 0\n",
            "episode: 143/360, score: 210.0, e: 0\n",
            "episode: 144/360, score: 210.0, e: 0\n",
            "episode: 145/360, score: 210.0, e: 0\n",
            "episode: 146/360, score: 210.0, e: 0\n",
            "episode: 147/360, score: 210.0, e: 0\n",
            "episode: 148/360, score: 210.0, e: 0\n",
            "episode: 149/360, score: 210.0, e: 0\n",
            "episode: 150/360, score: 210.0, e: 0\n",
            "episode: 151/360, score: 210.0, e: 0\n",
            "episode: 152/360, score: 210.0, e: 0\n",
            "episode: 153/360, score: 210.0, e: 0\n",
            "episode: 154/360, score: 210.0, e: 0\n",
            "episode: 155/360, score: 210.0, e: 0\n",
            "episode: 156/360, score: 210.0, e: 0\n",
            "episode: 157/360, score: 210.0, e: 0\n",
            "episode: 158/360, score: 210.0, e: 0\n",
            "episode: 159/360, score: 210.0, e: 0\n",
            "episode: 160/360, score: 210.0, e: 0\n",
            "episode: 161/360, score: 210.0, e: 0\n",
            "episode: 162/360, score: 210.0, e: 0\n",
            "episode: 163/360, score: 210.0, e: 0\n",
            "episode: 164/360, score: 210.0, e: 0\n",
            "episode: 165/360, score: 210.0, e: 0\n",
            "episode: 166/360, score: 210.0, e: 0\n",
            "episode: 167/360, score: 210.0, e: 0\n",
            "episode: 168/360, score: 210.0, e: 0\n",
            "episode: 169/360, score: 210.0, e: 0\n",
            "episode: 170/360, score: 210.0, e: 0\n",
            "episode: 171/360, score: 210.0, e: 0\n",
            "episode: 172/360, score: 210.0, e: 0\n",
            "episode: 173/360, score: 210.0, e: 0\n",
            "episode: 174/360, score: 210.0, e: 0\n",
            "episode: 175/360, score: 210.0, e: 0\n",
            "episode: 176/360, score: 210.0, e: 0\n",
            "episode: 177/360, score: 210.0, e: 0\n",
            "episode: 178/360, score: 210.0, e: 0\n",
            "episode: 179/360, score: 210.0, e: 0\n",
            "episode: 180/360, score: 210.0, e: 0\n",
            "episode: 181/360, score: 210.0, e: 0\n",
            "episode: 182/360, score: 210.0, e: 0\n",
            "episode: 183/360, score: 210.0, e: 0\n",
            "episode: 184/360, score: 210.0, e: 0\n",
            "episode: 185/360, score: 210.0, e: 0\n",
            "episode: 186/360, score: 210.0, e: 0\n",
            "episode: 187/360, score: 210.0, e: 0\n",
            "episode: 188/360, score: 210.0, e: 0\n",
            "episode: 189/360, score: 210.0, e: 0\n",
            "episode: 190/360, score: 210.0, e: 0\n",
            "episode: 191/360, score: 210.0, e: 0\n",
            "episode: 192/360, score: 210.0, e: 0\n",
            "episode: 193/360, score: 210.0, e: 0\n",
            "episode: 194/360, score: 210.0, e: 0\n",
            "episode: 195/360, score: 210.0, e: 0\n",
            "episode: 196/360, score: 210.0, e: 0\n",
            "episode: 197/360, score: 210.0, e: 0\n",
            "episode: 198/360, score: 210.0, e: 0\n",
            "episode: 199/360, score: 210.0, e: 0\n",
            "episode: 200/360, score: 210.0, e: 0\n",
            "episode: 201/360, score: 210.0, e: 0\n",
            "episode: 202/360, score: 210.0, e: 0\n",
            "episode: 203/360, score: 210.0, e: 0\n",
            "episode: 204/360, score: 210.0, e: 0\n",
            "episode: 205/360, score: 210.0, e: 0\n",
            "episode: 206/360, score: 210.0, e: 0\n",
            "episode: 207/360, score: 210.0, e: 0\n",
            "episode: 208/360, score: 210.0, e: 0\n",
            "episode: 209/360, score: 210.0, e: 0\n",
            "episode: 210/360, score: 210.0, e: 0\n",
            "episode: 211/360, score: 210.0, e: 0\n",
            "episode: 212/360, score: 210.0, e: 0\n",
            "episode: 213/360, score: 210.0, e: 0\n",
            "episode: 214/360, score: 210.0, e: 0\n",
            "episode: 215/360, score: 210.0, e: 0\n",
            "episode: 216/360, score: 210.0, e: 0\n",
            "episode: 217/360, score: 210.0, e: 0\n",
            "episode: 218/360, score: 210.0, e: 0\n",
            "episode: 219/360, score: 210.0, e: 0\n",
            "episode: 220/360, score: 210.0, e: 0\n",
            "episode: 221/360, score: 210.0, e: 0\n",
            "episode: 222/360, score: 210.0, e: 0\n",
            "episode: 223/360, score: 210.0, e: 0\n",
            "episode: 224/360, score: 210.0, e: 0\n",
            "episode: 225/360, score: 210.0, e: 0\n",
            "episode: 226/360, score: 210.0, e: 0\n",
            "episode: 227/360, score: 210.0, e: 0\n",
            "episode: 228/360, score: 210.0, e: 0\n",
            "episode: 229/360, score: 210.0, e: 0\n",
            "episode: 230/360, score: 210.0, e: 0\n",
            "episode: 231/360, score: 210.0, e: 0\n",
            "episode: 232/360, score: 210.0, e: 0\n",
            "episode: 233/360, score: 210.0, e: 0\n",
            "episode: 234/360, score: 210.0, e: 0\n",
            "episode: 235/360, score: 210.0, e: 0\n",
            "episode: 236/360, score: 210.0, e: 0\n",
            "episode: 237/360, score: 210.0, e: 0\n",
            "episode: 238/360, score: 210.0, e: 0\n",
            "episode: 239/360, score: 210.0, e: 0\n",
            "episode: 240/360, score: 210.0, e: 0\n",
            "episode: 241/360, score: 210.0, e: 0\n",
            "episode: 242/360, score: 210.0, e: 0\n",
            "episode: 243/360, score: 210.0, e: 0\n",
            "episode: 244/360, score: 210.0, e: 0\n",
            "episode: 245/360, score: 210.0, e: 0\n",
            "episode: 246/360, score: 210.0, e: 0\n",
            "episode: 247/360, score: 210.0, e: 0\n",
            "episode: 248/360, score: 210.0, e: 0\n",
            "episode: 249/360, score: 210.0, e: 0\n",
            "episode: 250/360, score: 210.0, e: 0\n",
            "episode: 251/360, score: 210.0, e: 0\n",
            "episode: 252/360, score: 210.0, e: 0\n",
            "episode: 253/360, score: 210.0, e: 0\n",
            "episode: 254/360, score: 210.0, e: 0\n",
            "episode: 255/360, score: 210.0, e: 0\n",
            "episode: 256/360, score: 210.0, e: 0\n",
            "episode: 257/360, score: 210.0, e: 0\n",
            "episode: 258/360, score: 210.0, e: 0\n",
            "episode: 259/360, score: 210.0, e: 0\n",
            "episode: 260/360, score: 210.0, e: 0\n",
            "episode: 261/360, score: 210.0, e: 0\n",
            "episode: 262/360, score: 210.0, e: 0\n",
            "episode: 263/360, score: 210.0, e: 0\n",
            "episode: 264/360, score: 210.0, e: 0\n",
            "episode: 265/360, score: 210.0, e: 0\n",
            "episode: 266/360, score: 210.0, e: 0\n",
            "episode: 267/360, score: 210.0, e: 0\n",
            "episode: 268/360, score: 210.0, e: 0\n",
            "episode: 269/360, score: 210.0, e: 0\n",
            "episode: 270/360, score: 210.0, e: 0\n",
            "episode: 271/360, score: 210.0, e: 0\n",
            "episode: 272/360, score: 210.0, e: 0\n",
            "episode: 273/360, score: 210.0, e: 0\n",
            "episode: 274/360, score: 210.0, e: 0\n",
            "episode: 275/360, score: 210.0, e: 0\n",
            "episode: 276/360, score: 210.0, e: 0\n",
            "episode: 277/360, score: 210.0, e: 0\n",
            "episode: 278/360, score: 210.0, e: 0\n",
            "episode: 279/360, score: 210.0, e: 0\n",
            "episode: 280/360, score: 210.0, e: 0\n",
            "episode: 281/360, score: 210.0, e: 0\n",
            "episode: 282/360, score: 210.0, e: 0\n",
            "episode: 283/360, score: 210.0, e: 0\n",
            "episode: 284/360, score: 210.0, e: 0\n",
            "episode: 285/360, score: 210.0, e: 0\n",
            "episode: 286/360, score: 210.0, e: 0\n",
            "episode: 287/360, score: 210.0, e: 0\n",
            "episode: 288/360, score: 210.0, e: 0\n",
            "episode: 289/360, score: 210.0, e: 0\n",
            "episode: 290/360, score: 210.0, e: 0\n",
            "episode: 291/360, score: 210.0, e: 0\n",
            "episode: 292/360, score: 210.0, e: 0\n",
            "episode: 293/360, score: 210.0, e: 0\n",
            "episode: 294/360, score: 210.0, e: 0\n",
            "episode: 295/360, score: 210.0, e: 0\n",
            "episode: 296/360, score: 210.0, e: 0\n",
            "episode: 297/360, score: 210.0, e: 0\n",
            "episode: 298/360, score: 210.0, e: 0\n",
            "episode: 299/360, score: 210.0, e: 0\n",
            "episode: 300/360, score: 210.0, e: 0\n",
            "episode: 301/360, score: 210.0, e: 0\n",
            "episode: 302/360, score: 210.0, e: 0\n",
            "episode: 303/360, score: 210.0, e: 0\n",
            "episode: 304/360, score: 210.0, e: 0\n",
            "episode: 305/360, score: 210.0, e: 0\n",
            "episode: 306/360, score: 210.0, e: 0\n",
            "episode: 307/360, score: 210.0, e: 0\n",
            "episode: 308/360, score: 210.0, e: 0\n",
            "episode: 309/360, score: 210.0, e: 0\n",
            "episode: 310/360, score: 210.0, e: 0\n",
            "episode: 311/360, score: 210.0, e: 0\n",
            "episode: 312/360, score: 210.0, e: 0\n",
            "episode: 313/360, score: 210.0, e: 0\n",
            "episode: 314/360, score: 210.0, e: 0\n",
            "episode: 315/360, score: 210.0, e: 0\n",
            "episode: 316/360, score: 210.0, e: 0\n",
            "episode: 317/360, score: 210.0, e: 0\n",
            "episode: 318/360, score: 210.0, e: 0\n",
            "episode: 319/360, score: 210.0, e: 0\n",
            "episode: 320/360, score: 210.0, e: 0\n",
            "episode: 321/360, score: 210.0, e: 0\n",
            "episode: 322/360, score: 210.0, e: 0\n",
            "episode: 323/360, score: 210.0, e: 0\n",
            "episode: 324/360, score: 210.0, e: 0\n",
            "episode: 325/360, score: 210.0, e: 0\n",
            "episode: 326/360, score: 210.0, e: 0\n",
            "episode: 327/360, score: 210.0, e: 0\n",
            "episode: 328/360, score: 210.0, e: 0\n",
            "episode: 329/360, score: 210.0, e: 0\n",
            "episode: 330/360, score: 210.0, e: 0\n",
            "episode: 331/360, score: 210.0, e: 0\n",
            "episode: 332/360, score: 210.0, e: 0\n",
            "episode: 333/360, score: 210.0, e: 0\n",
            "episode: 334/360, score: 210.0, e: 0\n",
            "episode: 335/360, score: 210.0, e: 0\n",
            "episode: 336/360, score: 210.0, e: 0\n",
            "episode: 337/360, score: 210.0, e: 0\n",
            "episode: 338/360, score: 210.0, e: 0\n",
            "episode: 339/360, score: 210.0, e: 0\n",
            "episode: 340/360, score: 210.0, e: 0\n",
            "episode: 341/360, score: 210.0, e: 0\n",
            "episode: 342/360, score: 210.0, e: 0\n",
            "episode: 343/360, score: 210.0, e: 0\n",
            "episode: 344/360, score: 210.0, e: 0\n",
            "episode: 345/360, score: 210.0, e: 0\n",
            "episode: 346/360, score: 210.0, e: 0\n",
            "episode: 347/360, score: 210.0, e: 0\n",
            "episode: 348/360, score: 210.0, e: 0\n",
            "episode: 349/360, score: 210.0, e: 0\n",
            "episode: 350/360, score: 210.0, e: 0\n",
            "episode: 351/360, score: 210.0, e: 0\n",
            "episode: 352/360, score: 210.0, e: 0\n",
            "episode: 353/360, score: 210.0, e: 0\n",
            "episode: 354/360, score: 210.0, e: 0\n",
            "episode: 355/360, score: 210.0, e: 0\n",
            "episode: 356/360, score: 210.0, e: 0\n",
            "episode: 357/360, score: 210.0, e: 0\n",
            "episode: 358/360, score: 210.0, e: 0\n",
            "episode: 359/360, score: 210.0, e: 0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xu5YIAxuUmBx",
        "colab_type": "text"
      },
      "source": [
        "**Results**  \n",
        "Here is a graph of the results. If everything was done correctly you should see the rewards over the red line.  \n",
        "\n",
        "Black: This is the 100 episode rolling average  \n",
        "Red: This is the \"solved\" line at 195  \n",
        "Blue: This is the reward for each episode  \n",
        "Green: This is the value of epsilon scaled by 200  \n",
        "Yellow: This is where the tests started."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tmrdKzWkUmB0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "0def6e2d-a74a-46dc-bcbe-b1f97e18665e"
      },
      "source": [
        "rolling_average = np.convolve(rewards, np.ones(100)/100)\n",
        "\n",
        "plt.plot(rewards)\n",
        "plt.plot(rolling_average, color='black')\n",
        "plt.axhline(y=195, color='r', linestyle='-') #Solved Line\n",
        "#Scale Epsilon (0.001 - 1.0) to match reward (0 - 200) range\n",
        "eps_graph = [200*x for x in epsilons]\n",
        "plt.plot(eps_graph, color='g', linestyle='-')\n",
        "#Plot the line where TESTING begins\n",
        "plt.axvline(x=TRAIN_END, color='y', linestyle='-')\n",
        "plt.xlim( (0,EPISODES) )\n",
        "plt.ylim( (0,220) )\n",
        "plt.show()\n",
        "\n",
        "\n",
        "envCartPole.close()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9b34/9d7ZrIAgbCEfZE1CGJZRFxAcKMq1AoKKnWtRWtdWq2/q/TWra1+rb1a295iFeuCFUJkUbmKVpRNqKBhkcWwC0IIJBC2LCSZzOf3x5xJJpN9MsmZ5f18PPKYmTNnzrxzIOd9PrsYY1BKKRV7HHYHoJRSyh6aAJRSKkZpAlBKqRilCUAppWKUJgCllIpRLrsDAEhJSTG9e/e2O4yYUli4A4CWLQfaHIlSKljr168/aozpGOznwyIB9O7dm4yMDLvDiCkbN14KwPDhK2yNQykVPBHZ35jPaxWQUkrFKE0ASikVozQBKKVUjNIEoJRSMUoTgFJKxShNAEopFaM0ASilVIwKi3EAqnktXH+Q+JOFACz/dIfN0dij4NRxNq5YQv7xY3aHopRtNAHEmDOlZTwy/xtmjCoC4H+/3m1zRM2r9PghCnf8h5PrFuEpOmVtFVtjUio4jV/LRRNAjDl9xg1A7w6t6Nwmke+un2hzRM3nySef5JlZz2CMYdy4cbz00ksMGzYMEU0AKvJ8/vnnXHnllY06hrYBxJiCYm8CcDpi66K3YMEC/vCHP3Drrbeyf/9+VqxYwfDhw/XiryLWuHHjGn0MLQHEmPwYTABvvvkm999/PxdccAGvv/46cXFxdoekVKO5XI2/fGsJIMaUJ4AYufPds2cP9957L0OHDmXhwoV68VfKj5YAYkwsVQEVFhYyefJkEhMTWbhwId26dbM7JKXCiiaAGBMrVUDGGB599FG2bNnCJ598ohd/papRZxWQiPQUkeUi8q2IbBORX1nb24vIUhHZZT22s7aLiPxNRHaLyGYRGdHUv4Sqv1hIAMYYfvvb3zJz5kweeughrrrqKrtDUios1acNwA08YowZDFwI3C8ig4EZwOfGmAHA59ZrgGuAAdbPPcA/Qh61CpqvCsgRxQng17/+Nc899xx33303L774ot3hKBW26kwAxphsY8wG6/lpIBPoDlwHzLZ2mw1Msp5fB7xtvNYCbUWka8gjV0HJLy5DJHobgXNycpg5cyZ33nknr776Kg6H9nNQqiYN+usQkd7AcGAd0NkYk229dRjobD3vDhzw+9hBa1vgse4RkQwRycjNzW1g2CpY+WfctIqP3qafmTNnUlpayqOPPqp9/JWqQ70TgIgkAQuBh4wxp/zfM8YYGjgu2Rgzyxgz0hgzsmPHoNc0Vg1UUOwmKSE6E8D27dt58cUXmTJlCoMGDbI7HKXCXr0SgIjE4b34zzHGLLI2H/FV7ViPOdb2LKCn38d7WNtUGMgvcdMqwWl3GCFnjOHWW2+lZcuWvPDCC3aHo1REqE8vIAFeBzKNMX/2e2sxcIf1/A7gA7/tt1u9gS4ETvpVFVVrV94uCkoKGhy8argzJWW0iI++BLBmzRrWr1/PH/7wB8466yy7w1EqIoi39qaWHUTGAF8AWwCPtfm/8bYDvAv0AvYDNxpj8qyE8XfgaqAQ+KkxJqPW7+gm5vA5F9G5NL4xv4uqh8zsU5QZQ+l/fwfA8LeG2RxRaGz79luO5+Vx0UUX4XRGX4JTqjqycuV6Y8zIYD9fZ2WwMWY1Nc+Xe0U1+xvg/oYGUuT0QGlDP6UaymNAomz64+LiYo7m5tK9Rw+9+CvVAGHTGnjmnbcg5Wy7w4h6T7y8hlYJLn499DHvhhUrbI0nFJ598kmeWbeO3StXQt++doejVPNpZE+3sOkkXVRaZHcIUSvrRBFPfbCVEreH0jJDnDNs/tkbrbi4mFdffZWJEyfSVy/+SjVI2JQAityaAJrKi//ewaKNWYzq04HSMg/xUZQA3n33XXJycnjggQfsDkWpiBM2VwItATSdDknexvV9xwoocXuIc4XNP3ujeDwenn/+eQYPHsz48ePtDkepiKMlgBjQsXUCAPuPFVBS5iHOGR2NwIsXL2bbtm288847OuWDUkEIm78aLQE0ndIyb1ff/ccKo6YKKDc3l8cee4x+/fpx00032R2OUhFJSwAxoLi0DICTRaVR0wj8u9/9jn379rF06dKQLI2nVCwKmyvBGfcZu0OIWsVu7/i9M6VllLo9xEd4G0BhYSHvvPMOU6dOZezYsXaHo1TECpsrgVYBNZ2KBOChuMwT8SWA9PR0Tp48yd133213KEpFtLC5EmgVUNMpdnurgM64y6w2gMhuBH755ZcZPHiw3v0r1UjhkwC0BNBkiku9JYCCYjfGENElgK+//pqMjAzuu+8+ne9fqUYKiyuBiGgJoAn5qoB8vYEieRzAyy+/TFJSErfddpvdoSgV8cLiSuAQh5YAmpAvAfhEagng2LFjzJs3j9tuu402bdrYHY5SES8srgQOcWgJoAn52gB8IrUX0JtvvsmZM2f4xS9+YXcoSkWFsLgSOMSh3UCbUGAJIBIbgYuKivjb3/7GJZdcwrnnnmt3OEpFhbAYQaNtAE0rGqqAXnnlFQ4cOMDs2bPtDkWpqFGfJSHfEJEcEdnqty1dRDZZP/tEZJO1vbeIFPm990q9gtA2gJBzl3kY/+eVfPbtkfKRwD6RlgCMMbz88suMHj2ayy67zO5wlIoa9SkBvIV3ice3fRuMMeWTr4jIi8BJv/33GGMatM6gQxwUlOqawKGUV1DCrpx8ZizaQptEF0kJLvKL3YCVADx1HCCMfPTRR+zevZunnnrK7lCUiip13goaY1YBedW9Z63/eyOQ1pggnOLkxJkTjTmEClRezW8odntIbhFX/lZCBDUCG2P43e9+R58+fXTSN6VCrLFXgkuAI8aYXX7b+ojIRhFZKSKX1PRBEblHRDJEJMNd4uZ40fFGhqL8+db9NcbbC8g/AURSFdDHH39MRkYGjz/+OHFxcXV/QClVb41tBJ5G5bv/bKCXMeaYiJwHvC8i5xhjTgV+0BgzC5gF0Dm1szl+RhNAKPkGyRqgpNRDmxYV/9SRtB7AX//6V3r06KEDv5RqAkHfCoqIC7geSPdtM8YUG2OOWc/XA3uA1LqO5RIX+SX5uD3uYMNRNfCYqlVAkTISeMuWLXz66adMnz5d7/6VagKNuRJcCWw3xhz0bRCRjiLitJ73BQYAe+s6kNPhBNB2gBDyGO+0Dx6PoaSscgKIlAVhfvGLX9CxY0fuv/9+u0NRKirVpxtoGvAlMFBEDorIz6y3bqZq4+9YYLPVLXQBcK8xptoGZH8uh7d6QtsBQse6/pePAYi0NoCvvvqKNWvW8MQTT5CSkmJ3OEpFpTrbAIwx02rYfmc12xYCCxsahNNbaEDbAULHVwJwe7yPrRMrEkBinINCW6Kqv5kzZ5KUlMQdd9xhdyhKRa2wGAnsqwLSEkDoWNd9yqwnrRJc/GnKDzh9xk2v9i3J+97G4Opw9OhR0tPTueuuu3TSN6WaUFgkgPIqIC0BhIzHlwEsToEbR/a0KZqGef311ykuLta6f6WaWFhUBvtKAHlFdTYXqHoyla//OB2R0fXT4/Hw6quvMm7cOM455xy7w1EqqoVFAnA5XAhCTkGO3aFEDU9ABnBESAL44osv+O6775g+fbrdoSgV9cIiAQhCSssUjuQfsTuUqBGYAJwRsnzi22+/TVJSEpMnT7Y7FKWiXlgkAIAuSV04XHDY7jCihicCq4AKCwuZP38+U6ZMoVWrVnaHo1TUC5sE0DmpM4fzNQGEigksAURAAnjvvfc4ffq0dv1UqpmETQLoktRFE0AIRWIJYPbs2Zx11lmMHTvW7lCUiglhkwA6t+rMkfwjVe5cVXCqNAKHeRvAwYMH+eyzz7j99ttxOMLmv6VSUS1s/tK6JHWhyF3EqeIqE4eqIFRpBA7zEsCcOXMwxnD77bfbHYpSMSNsEkBKS+98L8eKjtkcSXSIpHEAxhhmz57N6NGj6d+/v93hKBUzwiYBdGjRAYBjhZoAglXsLiuvQiurMhI4fBNARkYGmZmZevevVDMLnwTQ0koAWgIISn6xm4GPf8JfP/cuzhZJVUCzZ88mISGBG2+80e5QlIop4ZMArBKATgcRnAJrwfe567yzvAX2AgrXkcDFxcWkpaUxadIk2rZta3c4SsWU8EkALbUKqDF8c/z75v+vMg4gTKuAlixZQl5envb9V8oG9VkQ5g0RyRGRrX7bnhaRLBHZZP1M8HvvNyKyW0R2iMhV9Q2kbaL37k+rgILju+AXu8uAyBkHMHv2bLp06cL48ePtDkWpmFOfEsBbwNXVbH/JGDPM+lkCICKD8a4Udo71mZd9S0TWxeVw0TaxrVYBBckTsAJYJLQB5Obm8tFHH3HrrbficoXFzORKxZQ6E4AxZhVQ36vydcA8a3H474DdwKj6BtOhRQctAQTJ4L3g+677VRNAc0dUt7S0NNxut1b/KGWTxlwWHhCRzVYVUTtrW3fggN8+B61t9dKhZQdtAwhSYL//wNfhOBJ49uzZDB8+nCFDhtgdilIxKdgE8A+gHzAMyAZebOgBROQeEckQkYzc3FwAOrbsqGsCBCnwgh/uVUBbt25lw4YNevevlI2CSgDGmCPGmDJjjAd4jYpqnizAf93BHta26o4xyxgz0hgzsmPHjgB0a92NQ6cPBRNSzPO/4J8sLA37RuC3334bl8vFT37yE7tDUSpmBZUARKSr38vJgK+H0GLgZhFJEJE+wADgq/oet2tSV3IKcnB73MGEFdP8r/dDf/9pWJcA3G4377zzDhMmTMCX/JVSza/OrhcikgZcCqSIyEHgKeBSERmG97qzD/g5gDFmm4i8C3wLuIH7jTFl9Q2mW+tuGAxH8o/QvU29mw4UVReBD+dxAMuWLSM7O1unflDKZnUmAGPMtGo2v17L/s8CzwYTTNfW3oJFdn62JoBG8ngqvw6nkcBz5swhOTmZiRMn2h2KUjEtrDoHdmvdDUDbAYIQWOUTrmsCFxYWsmjRIqZMmUJiYqLd4SgV08IqAXRNskoAp7NtjiTyVO0FVPl1uLQBfPjhh+Tn53PLLbfYHYpSMS+sEkDnpM4IoiWAWuScOkPu6eIq2wPv+MN1TeA5c+bQrVs3XfZRqTAQVgnA5XDRqVUnTQC1GPX/Puf8Zz+rsj1wIc1wLAHk5eXx8ccfM23aNJzOes0QopRqQmGVAMDbDpCdr1VADRV4xx+OawIvWLCA0tJSrf5RKkyEXQLo2rqrlgCCEAkjgefMmcPZZ5/NsGHD7A5FKUUYJoBuSVoCCEZglU+4rQn8/fffs2rVKm655RYkDEojSql6jANobl1bd+VI/hHcHjcuR9iFF7Z8s4G2SXRhCL8SwLx58wB06gelwkjYlQC6JHXBYMgtyLU7lIjiG/iV3DKOMo+p2ghs8133nDlzuPDCC+nbt6+tcSilKoRdAmiX6J1Z+mTxSZsjiSy+O/44pwN3manaCGzjv/TWrVvZvHmzNv4qFWbCLgH4loY8ceaEzZFEpninA7fHU6VXkMvGDDB37lycTic33nijbTEoparSBBAl/EsAHgOrdh6t9L5dTQAej4e5c+cyfvx4OnXqZE8QSqlqhV0CSE5MBuDkGa0Cqq81u4+SX+ydQtvl9F7pP9pS0ZPKIdjW8+Y///kP+/fv1+ofpcJQ2CUALQE0zKkzpdz6+jre3+hddycuzBb/nTt3Li1atGDSpEl2h6KUChBeVwsgOcFbAtAEUD8lbg/GQEGJd9mF+DBKAKWlpbz77rtcd911JCUl2R2OUipA+FwtLC3jWuJyuLQXUD356v5L3d5+oHHO8Blk9e9//5tjx45p9Y9SYarOBCAib4hIjohs9dv2PyKyXUQ2i8h7ItLW2t5bRIpEZJP180pDAxIR2ia21RJAPfk6+7itjv+uakoAdtX/z507lw4dOnDVVVfZ8v1KqdrVpwTwFnB1wLalwBBjzA+AncBv/N7bY4wZZv3cG0xQyQnJmgDqqbwEUOYtAYRLFVB+fj4ffPABU6dOJS4uzu5wlFLVqPNqYYxZBeQFbPvUGONbuX0t0COUQbVNbKtVQPXkG/HrSwDVVQEFjgloDu+//z6FhYVa/aNUGAvF7eJdwMd+r/uIyEYRWSkil9T0IRG5R0QyRCQjN7fytA9aBVR/vsXg3WUV4wDCwdy5cznrrLO4+OKL7Q5FKVWDRl0tROS3gBuYY23KBnoZY4YDvwbmikib6j5rjJlljBlpjBnZsWPHSu8lJybrOIB68t3cl9bSBtDccnJy+PTTT5k2bRoOO+egUErVKui/ThG5E/gRcIux6hiMMcXGmGPW8/XAHiC1ocdum6AlgPoK7AUUX00VUHM3As+fP5+ysjKd+VOpMBdUAhCRq4FHgR8bYwr9tncUEaf1vC8wANjb0ONrFVD9+RKA2+NrA6j6T1oWODVoE0tLS2PIkCGce+65zfq9SqmGqU830DTgS2CgiBwUkZ8BfwdaA0sDunuOBTaLyCZgAXCvMSav2gPXIjkxmYLSAtwed907x7iKRmCrDcBlb5XL/v37WbNmDdOmTbM1DqVU3epcccUYU91f8us17LsQWNjYoHzTQZw8c5IOLTs09nBRzQR0A7W7Edi38MvNN99saxxKqbqFZQudzgdUf1W6gdq88ldaWpou/KJUhAjLBOCbD0jHAtStvA0gDKqAMjMz+eabb7T6R6kIEZYJQEsA9Rc4EtjOKqC0tDQcDocu/KJUhAjLVdc1AdTNGIOIVJkLKLAb6N2X9KGotKxZ4klLS+Oyyy6jS5cuTf59SqnGC8sEoIvC1K3MY3A5pUoJIHAg2G8nDm6WeNavX8/u3buZMWNGs3yfUqrxtAooQvkaf6t0A7WpCmju3LnEx8dz/fXX2/L9SqmGC8sE0CahDYJoI3AtfHf+noCJ3uxYD6CsrIz09HSuueYa2rVr1+zfr5QKTlgmAIc4aJ3QWksAtfCN7g2c6dOO6aC/+OILDh06pL1/lIowYZkAQKeDqEtZeQmg8nY7JoNLS0ujVatWXHvttc3+3Uqp4IVtAkhOSNYqoFoYb5tv+XTQPs1dBVRSUsKCBQu47rrraNmyZbN+t1KqccI2AWgJoHY1lQCauwro008/JS8vT2f+VCoCaQIIM2Uewx8+/Jbsk0V17gfVrPbVzG3AaWlptG/fnvHjxzfvFyulGi1sE0CsLgrz9b48Xl/9Hf81f3Ot+3lqKAHQjDM/FxYW8sEHHzBlyhTi4+Ob74uVUiERtgkgVheFCZzfv679AruBdklO5MaRIV2iuUb/93//R0FBgfb+USpChW8CsBaGt2NBc1vV89f1VQEFJgCnQ/jTlKGhjqpaaWlpdOvWjUsuqXHpZ6VUGAvbBJCcmIzHeMgvybc7FFtINZX5/snQV0AIzI+OZlr+8fjx4yxZsoSbbroJp9PZLN+plAqteiUAEXlDRHJEZKvftvYislREdlmP7aztIiJ/E5HdIrJZREYEE1hKyxQAjhQcCebjUcm/vr+mKqDmsmjRIkpLS7X3j1IRrL4lgLeAqwO2zQA+N8YMAD63XgNcg3ct4AHAPcA/gglsQPsBAOw6tiuYj0es2i7n/hf7mrqBOpppQZi0tDT69+/Peeed1yzfp5QKvXolAGPMKiBwbd/rgNnW89nAJL/tbxuvtUBbEena0MBSO6QCsPPYzoZ+NCpUV5Pjf7PvqaENoDku/4cPH2b58uVMmzYNaaYqJ6VU6DWmDaCzMSbben4Y6Gw97w4c8NvvoLWtEhG5R0QyRCQjNze3ysFTWqbQNrFtzCaA6lRXAghsJG+ONoB3330Xj8ejvX+UinAhaQQ23qtQgyqjjTGzjDEjjTEjO3bsWOV9ESG1Qyo782IrAdRWpV+5BGA9Bo4Da4Yb8rlz5zJ06FAGDRrU9F+mlGoyjUkAR3xVO9ZjjrU9C+jpt18Pa1uDpXZIjdkSQLVVQH451lNeAqj7c6G0d+9e1q1bp42/SkWBxiSAxcAd1vM7gA/8tt9u9Qa6EDjpV1XUIKntU/n+5PcUldY+LUKs8L/br2kcQHXdR0Np3rx5ANx8881N+j1KqaZX326gacCXwEAROSgiPwP+CIwXkV3AldZrgCXAXmA38BpwX7DB+RqCd+ftDvYQEcfUUpPmf7GvqRtoU3cCSktLY/To0fTq1atpv0gp1eTqtSawMaam1r4rqtnXAPc3Jigf/55A53Y+NxSHDHu+63n1A8EqntdcBdR0GWDLli1s3bqVv//97032HUqp5hO2I4EBBnTwjgWIpXaAslpagf17/JSVNwI3XwkgLS0Np9PJ1KlTm+5LlFLNJqwTQFJ8Et1ad4upnkCBC7xUeq/aNoDK+zRVG4Axhnnz5nHllVfSqVOnJvkOpVTzqlcVkJ1irSeQ78Je/UAwU+V5lUZgK6XPnX4BZ9xlIYtr3bp1fPfddzz11FMhO6ZSyl7hnwDap7Jo+yK7w2g2NdXte9+reF7TQDBf3ri4f0pI40pLSyMhIYHJkyeH9LhKKfuEdRUQeEsARwuPklcUOBNFdAqs0nk4fRNPvO+dg69yG0ANcwE1QSNwWVkZ6enpTJw4kTZt2oT8+Eope0REAoDYaQgOrAJ6b2MW/1q7H6g81NqXC6pUATVBE8Dy5cs5cuSITv2gVJTRBBBmapve2WNTCSAtLY3WrVszceLEkB9bKWWfsE8Afdr1wSnOmEkAZfXtBVRDG0CoFRcXs3DhQiZPnkyLFi2a9LuUUs0r7BNAvDOePu36aAIgcEUwU+nRJ9QlgE8++YSTJ09q9Y9SUSjsEwDA2Slnsy13m91hNIvaqoAqjwSu/OgT6oFgaWlppKSkcMUVVQZ9K6UiXEQkgOFdhrP96HYKSwvtDqXJ+Ub4Vqf6FcECG4FDlwHy8/NZvHgxU6dOJS4uLmTHVUqFh4hIACO6jsBjPGw+stnuUJqc78Je3YU8cEWwDzZl8cxHmZX2CWUJYPHixRQVFWn1j1JRKiISwHldvevOrj+03uZImp6vTn/Vzlx++uZXld8L6AX08vI9VT4fyhLAvHnz6N69O6NHjw7ZMZVS4SMiEkCPNj1IaZnChuwNdofS5PwbgZfvqLxUpqdSG4Cp0uc/lO2/J06c4JNPPuHGG2/E4YiI/yZKqQaKiL9sEWFE1xGsz46BEkCt3TqrrgfgL5Ttv++//z6lpaXcdNNNITyqUiqcREQCAG810LbcbZxxn7E7lCZV73EAnqpdPkPZBTQ9PZ3evXszatSokB1TKRVegk4AIjJQRDb5/ZwSkYdE5GkRyfLbPiEUgY7oOgK3x83WnK2hOFzYqm09gMBeQE1VBXTs2DE+++wzbrrppiZdYEYpZa+gE4AxZocxZpgxZhhwHlAIvGe9/ZLvPWPMklAEGisNwbWtB+CfG4pLy6pJAKG5WC9atAi3263VP0pFuVBVAV0B7DHG7A/R8aro3bY3bRPbRn1DcH3HARSWlFVZ/CVU9+rp6ekMGDCAYcOGheiISqlwFKoEcDOQ5vf6ARHZLCJviEi76j4gIveISIaIZOTm5la3S+D+MdEQXPuSkBXPi0rLqvT5D0UbwJEjR1i+fLlW/ygVAxqdAEQkHvgxMN/a9A+gHzAMyAZerO5zxphZxpiRxpiRHTt2rNd3jegygi05WygtK21s2GGrpiogY0zlBFBSVqXSPxTX6wULFuDxeLT6R6kYEIoSwDXABmPMEQBjzBFjTJkxxgO8BoSsG8nwrsMpKSsh82hm3TtHqJpKAG6PCagCclep8glFCSA9PZ3BgwczZMiQRh9LKRXeQpEApuFX/SMiXf3emwyErNvO8C7DAdiYvTFUhww7NZUACkvKqrQBBFYBNfbyn5WVxerVq/XuX6kY0agEICKtgPGA/6K9fxKRLSKyGbgMeLgx3+EvtUMqLeNaRnVDcE3jAIb+7lPyi93lr4tKyqrU0Te2ADB//nyMMZoAlIoRjVoU3hhTAHQI2HZboyKqhdPhZGjnoWw8HMUlgFoGAp8qqkgA3l5AlTW20TY9PZ1hw4YxcODARh1HKRUZImYksM/wLsPZdHgTHlNLf8kIVttUEL4qH4d4ewEFXu8bMxPo/v37Wbt2rd79KxVDIi4BjOg6gtMlp9mTV3UmzGhQ21QQpdZ7reJdNVQBBZ8B3n33XQBuvPHGoI+hlIosEZcAhne1GoKjtBqotnEAbmuUWKsEF4Wl1fUCCv5709PTOf/88+nbt2/wB1FKRZSISwDndDwHl8MVtT2BapsKorQ8ATitEkDgHsFlgN27d7N+/Xqt/lEqxkRcAkhwJTCk0xA2HI7OnkC1VgGVWVVACS6rG2jgbKDBfWd6ejqg1T9KxZqISwDgbQjemL0RU+vc+ZGptiogXwmgZbyTotKyKu8H2wSQnp7OxRdfTM+ePYM7gFIqIkVsAsgtzOXQ6UN2hxJytVUBua0SQFKCC2MqEoJPMCOBMzMz2bJli1b/KBWDIjIBjOg6AiBqBoSVuD1knywCoKyWQk2px1cC8A7fCCwFBFMASE9PR0SYOnVqEJ9WSkWyiEwAQ7sMxSEOvj70td2hhMRjCzdz0XPLKHaX1d4I7K5oAwBrQjg/wXQDTU9PZ9y4cXTt2rXunZVSUSUiE0BSfBIjuo5g5f6VdocSEp9uOwx4SwK1NQK7rRJAUoITgDOllauAGnr9LygoYPv27Vr9o1SMisgEAHDpWZey9uBaikqL7A6l0XyXfI+nrkZg73u+KqCCkoqpIRzS8DaAnJwcnE4nN9xwQ8MCVkpFhchNAL0vpaSshLUH19odSsiUejz1HgcAkH+mIgE4HdLgEkBOTg6XX3459V2PQSkVXSI2AYzpNQaHOFixb4XdoYSMu8zUqxuorw3A7ZcsnA5pUAng9OnTnDlzRqt/lIphEZsAkhOTGdF1BCv2r7A7lEbzXfNLy2pvAygfCBZfdRJXl8PRoF5Aubk5iMDkyZMbEqpSKopEbAKA6GoHgKqrflV5P6AE4K8hVUDGGHJycmnfvj3t27cPKlalVOQLxZrA+6wFYDaJSIa1rb2ILBWRXdZjtQvDN1a0tYDVZw8AABiUSURBVAO46ywBWAkg3lnlPZdD6t0NdO3atRQXF9OxY6fgAlVKRYVQlQAuM8YMM8aMtF7PAD43xgwAPrdeh5yvHSDSu4Maqx9QaZnBU8syB+XTQddQAqjvXEDp6ek4HEJKSkqDY1VKRY+mqgK6DphtPZ8NTGqKL0lOTGZIpyF8efDLpjh8s/HV+rg9nnpNB92iphJAPVoBPB4P8+fPp3379jidVY+jlIodoUgABvhURNaLyD3Wts7GmGzr+WGgc+CHROQeEckQkYzc3Nygv/yiHhex9uDaqFgh7GRRKev3H6+y/aWbhgIVcwE5HUJiXOV/Okc92wBWr17NoUOH6NRJq3+UinWhSABjjDEjgGuA+0VkrP+bxjtlZ5XbWmPMLGPMSGPMyMb0Qx/Tawynik+RcSgj6GOEi9te/6ra7ckt4gAosUoADpHywWA+9W0DmDdvHi1atKB9+w517quUim6NTgDGmCzrMQd4DxgFHBGRrgDWY05jv6cmP0r9EfHOeOZumdtUX9Hk6prUOsHlrarxNQIL0CKucvVNfdoAioqKSEtLY9KkSVr9o5RqXAIQkVYi0tr3HPghsBVYDNxh7XYH8EFjvqc2bRPbcm3qtaRtTcPtcdf9gTD315uH0aVNYqVt8S7vP5OvCshbAqh8AXc5HHUOBFu0aBEnTpxg+vTpIYxYKRWpGlsC6AysFpFvgK+Aj4wxnwB/BMaLyC7gSut1k7nl3FvIKchh6Z6lTfk1zaK6QV7xTu8/U3kJQKiSAOozDuCf//wnffv25dJLLw1JrEqpyFb1atMAxpi9wNBqth8DrmjMsRtiwoAJtElow3vb3+OaAdc019c2iXiXo8pgsIQ4XwLwbheBxIAqIJez9jaAnTt3smLFCp599lkcjoge/6eUCpGouBIkuBK4su+VLNm1JDKXifQLOcHlIHAsmK8E4JsOOrAKqH+nJG8JoJavmDVrFi6Xi7vuuitUUSulIlxUJACACf0nkHU6iy05W+wOpVHiXY4qSSzOSgA7j+QDlXsB3XJBLz779ThctTQCnzlzhjfffJNJkybRpUuXpgteKRVRoiYB+Kp+Pt71sc2RNE6Cy8l9l/UHoHWi9yIfWLMjUjEYzGld9R1ScxXQwoULycvL4+c//3kTRa2UikRRkwC6te7G0M5D+XDXh3aH0ijxLgc/G9OHfX+cyKNXDQSgXcv4Svv4NwL7ev4M7taGs7u0rvaYs2bNon///lx++eVNGLlSKtI0qhE43EwdPJXHlz/O7rzd9G/f3+5w6s34NQIkuCpy8m0X9ea2i3pXqRJyiFQpATx17TnVHnv79u2sWrWK559/Xht/lVKVRNUV4c5hd+IQB29sfMPuUILmnwB8Aqt2/AeCueoY/fXKK68QFxfHnXfeGaoQlVJRIqoSQPc23Zk4YCJvbnozYgeF+Ub91qW8CqiWBJCZmcnLL7/MT37yE537RylVRVQlAIDpI6ZzOP8wH+38yO5Q6s2/hichru5/klYJLlpYvYCcNTT8GmO49957SUpK4k9/+lNI4lRKRZeoSwATBkyga1JX/rnxn3aHEhRfn/+aLH14LIlxTlrG1V4CWL16NatWreKZZ57Ru3+lVLWiLgG4HC5+OuynLNm1hKxTWXaHUy/+awDUVqUDFSOAfVVA1bUBeDwennvuOZKTk7njjjuqvK+UUhCFCQDgruF34TEe3tr0lt2h1KnMY2jI4GVfAgjsBeRv5syZfPzxxzz99NO0atUqJHEqpaJPVCaAfu37cXmfy3l94+u2LRRT5jG8tmovhSW1N0b7JnirL9+F39cLKHAG0MOHD/PUU09x5ZVX8qtf/apBx1ZKxZaoTAAA94y4h+9OfMfCbxfa8v0fb83m2SWZvPDvnZW2ezyGV1bu4WRRKdDwBJBodRP1TQUR2GTw4IMPUlRUxN/+9rd6LxKvlIpNUZsApgyewpBOQ3h65dO2TBBXWFIGwImikkrbv96Xxx8/3s7IZ5aybu+x8jn+68tlXfErqoAq/gk3bdrEggULeOyxxxg0aFBjwldKxYCwSAAHjhdS4g5tVY3T4eTBUQ/ybe63bDq8KaTHro/ye++A67uvkbe0zHDTrLWUeoL7vX2NwE6/m/zf//73JCcn89BDDwV1TKVUbAmLBHCisJSj+cUhP+6UwVOId8bz/JrnQ37sutRU/eIrGfg0tATgk5KUwNjUjgzr1Q7w3v2/9957PPzww7Rt2zaoYyqlYkvQCUBEeorIchH5VkS2icivrO1Pi0iWiGyyfibU53g5p0OfANq3aM8TY58gfVs6n+39LOTHr43bqtsPvLwXFlduFD586kxQx493OXj7rlEM69mWLVu2cMMNN5CcnKwNv0qpemtMCcANPGKMGQxcCNwvIoOt914yxgyzfpbU52A5QV4I6/JfF/8XPdr04PFljzdrjyDfnX5g+0N+QALYfOBEo77n+PHjXH311RQWFjJ37ly9+1dK1VvQCcAYk22M2WA9Pw1kAt2DPV5uE1QBgXe1sGcue4Z1Wet4bf1rTfId1SkqtRKA37Ydh0/zl892AfDefRd7tx05HfR3HD9+nDvvvJPs7Gw+/PBDJkyoV2FLKaWAELUBiEhvYDiwztr0gIhsFpE3RKRdDZ+5R0QyRCQDIOdU0yQAgNuH3s5lvS/j0c8e5Uj+kSb7Hn8F1p2+fzfPa/++mqwTRQB0a9sCgKwT3pLPqN7t+e8JZ9fr2MYYFixYwJgxY/joo494+umnOe+880IZvlIqBjQ6AYhIErAQeMgYcwr4B9APGAZkAy9W9zljzCxjzEhjzEiXQ5qkDcAvRv4x8R8UlBTw7BfPNtn3+PNVARUUVzT6+vd06tDKu8jLISshPDt5CPeM7Vfj8R4Zn8qMa84mPz+fSZMmMXXqVPLz81m8eDFPPvlkU/wKSqko16gEICJxeC/+c4wxiwCMMUeMMWXGGA/wGjCqruO4HA5yT1e0AZR5DM8tyWRPbn5jwqtkYMpA7h5xNzO/nsmXB74M2XFr4hsB7CsJBLYFuJwOkhJc5QkgKbH2tXkevGIA947rxy233MJHH33ECy+8wN69e7XaRykVtMb0AhLgdSDTGPNnv+1d/XabDGyt61gup3A0v2LA1LZDJ3l11V7+a/43wYZXrefHP0+v5F7c+t6tnC4Ovu69PspLANbji5/urLJP60RX+X5JCbUngBMnTvDnP/+ZxYsX88wzz/DII4/gdNZv7QCllKpOY0oAo4HbgMsDunz+SUS2iMhm4DLg4boO5HQIJworEsA3Vs+YM6Wh7bXTJqEN/5r8L/ad2McvP/lltfvMXL6bdXuP1Xqc/+w5yh8/3l7prv7UmVJmLNxMrlWV5buwH8sv5rbX1/H35burxpMYB3jX+G0VX30CyM3N5YEHHqB9+/Y88sgjjB49ml/+svrYlVKqIYJeE9gYsxq/Aa9+6tXts1IQDuF4YWn56/X7jwNQUMdEasEY02sMv73kt/xh1R+4uMfF3H3e3eXvGWP4n3/vAGDfHydW+tz/fr6Lc3skc+nATvzx4+1sPniS3h1acvOoXgDMWrmXeV8foHvbFjx4xYDyKqCc08XknC4mtXMS7/78Iob9fmn5Mdu08J7+pHhXtdNAf/nll0ydOpWsrCzuvvtupk+fzvnnn69z/CilQiIsRgI7HcKpM6Xlg6f2HSsE4Pu8QooCRs6GwlPjnuKH/X7IAx8/wLLvlpVvr6nEUeL28OLSndz55tdAxRz8S7+t6FH01b48AE5Yk7wFjvh9ceowklvEVdrmKwG0rqb+/y9/+QsXX3wxpaWlfPXVV8yaNYtRo0bpxV8pFTJhkQBcDsEYymfIPFbgrUYxJviRsrVxOpzMvX4uA9oP4Nq0a/li/xdA5Ynb/Ltvfne0oPz5Yws2832et+F2Z85pK07DtqyTALy++js+3HyoyoCvDknxVS7ebayE4N8AXFhYyNSpU3n44Ye5/vrr2b17N+eff36jf2ellAoUFgnAN6OlrxroWH4JAzolWc+bpntoh5Yd+Oz2z+jZpicT5k5g7cG15QkIKl/0/QdrpWcc4Gh+MS3inBzIK+La/13NnHXflzf2AjwwdyP7jhbQs32L8m3trW6fT107mFduHQFU3PknJbgwxrBs2TJuuOEGFixYwBNPPMG7775L69atm+T3V0qpsEgAviqVE4UlFJWUUVhSRmpn74XPv3dQqHVJ6sKyO5bRJakLV79zNWsPfFX+3vyMA9zyz7Vc//IaXl6+G6dDeOunFXfilw/yrrO7Jeskf/nM28Pn52P7kmgt6u4xcPclfcv3963k9dPRfbh6iLejlK8KyH3sABMnTuSKK65gzZo1vPDCC/z+97/XXj5KqSYVdCNwKPmWNTxeWFpe/ZPauTUfbckmr6BxCeCJ97cy8QddubBvh2rf79a6G8tuX8bYt8by4NKptJbfk0Bf/rn6O4zxrrxVVFpG346tOL93+/LPPXxlKi3inGRmn2LboVMATB3Zg/sv78+MhZvp1DqRS1M7Aduq/d7Tp09TtO1z4tZ8zNL1y2jZIpHnnnuOhx56iMTExEb9zkopVR9hUQLwJYA/L93JC1YvnAGdvVVA/9lzlPvnbOBMafWNwVsOnuTutzOqXXrxRGEJ/1q7n2mvra31+3sm92TZ7ctIcLYkJ+Fppo5MwRjvaN1xqR0BGNi5Na0SXPx/P0zltdtH0r9TEi9MHcpFfomlR7uWtEmM4+VbzuPpH59D5+SESt9jjOGbb77hwQcfpHv37jz5yP0cz/ySG66fzKZNm5gxY4Ze/JVSzSYsSgBxLgcJLgeZ2afIzPbeTXdNTqR1gosPN2cDcPOonlwywHsx/ucXe/lg0yEeu/psbn3dO/3QwvUHWbghiwv7dmDGNWez/1gBN77qHfHr664/P+MAu3Py+c2Eqqtl9WnXh+mD/8rzG6ZymNnARIb3asdZKS0B6NXB+/jA5QMqfc63PSUpvryaxyfB5cRTXEDRvk1Mn/4en3zyCVlZWcTHx3PzzTdz3333ac8epZRtwiIBCN4FTnwTpQF0aJVAh6R4Tlu9aTYfPMmIXu24b84GVu7MBeDutzPK9//LZ7s4VlDCpgMn+Pe2w5UacQH+35JMZq3aC8BdY/rQuU3FnXb619/zzcGTtG8xhDbua/nX1n8w7QfduPvCX5BhjUnw1dcHSkny3uX7qpiMMezbt4+VK1fy1ltvkbV6NZ6yMhYkJzN+/HiuueYarr32Wjp27NiYU6aUUo0WFgkAoEtyYqUE0L1dC9q3ii8fE/DXz3fxxurvOFZQwvUjutMizsmcdd+T4HJgDBzzaysIvPgD5Rd/gMkz19AqwUW7lvH87JI+PLZwC+BdZrGP6xd06+9m3q4nuWRAB26/aDpH84u5/aKzqo370tSO/LiPg9TTG7j99r+wbNkysrKyAOjXrx8zHnuMa665hgsvvBCXK2xOt1JKhU8C+PONQ0n76gAeY5hyXg+cDuFnY/rSfdth4pzCog1ZHHOX8KcpP+DGkT05kFdIfrGbC/p0YOGGg6zff5xpo3qRkhRPj3Ytyi/qge4a3YfDp7yJZtXOo/z8X+tJSUrgumHdyD5ZxHlntecnFyzkpgU3cd+S+zhScISnfvRUeTWN2+3mm2++YfXq1XzxxResXr2aI0e8A8JSUlK4/PLLGTduHKNHj2bIkCHak0cpFbYkcJZKO4wcOdJkZGTU+L67zMP/LtvNxB90Le8e6u/zzCMs/uYQPx/bj8Hd2mCM4eUVe4h3Opj1xV5yTxfTvW0L/v6T4QzvVbE8wepdR5m//gB3XNybEb0qL1vg9ri55//u4c1Nb3J5p8sZfWI0X638ijVr1pCf752ltHfv3owZM6b8Z9CgQTgcYdGuXqeNGy8FYPjwFbbGoZQKnoisN8aMDPrzkZAAGuNEYQmvfbGXX14xgARX7XfjHo+HHTt2sHLlSr788ku++vortqdsh0uBAui7ri9XD72aSy65hDFjxtCjR48mibk5aAJQKvI1NgGETRVQU2nbMp7/uqrySlsnT55kx44d7Ny5s9Ljrl27KCz0tjl07tyZ888/n2nnT6PtoLb8z/f/Q/Y12fS+tDfXX3g98c54O34dpZQKmahNACUlJezdu7fKRX7Hjh3k5OSU7+dwOOjTpw8DBw7ksssu49xzz2Xs2LH069evUvfMmwtuZvri6Tz62aO8sv4Vnhz7JLf84BZcjqg9hUqpKBfxVUAnTpwgMzOTzMxMvv32W7Zv387OnTvZu3cvZWUVg8c6duzIwIEDSU1NrfTYr18/4uPrfze/ZNcSHl/2OBsPbyS1QyozRs9g0tmTaNei2qWPw5ZWASkV+WKiDcAYQ1ZWFtu3by+/2PueHz58uHy/xMTEKhf41NRUUlNTadcudBdoYwzvb3+fp1Y8xZacLTjFySVnXcJ1A6/jR6k/ol+7fmE/uEsTgFKRL2wTgIhcDfwVcAL/NMb8saZ9AxPAkSNHWLFiBcuXL2fDhg1s376d06crZuRMTk5m0KBBnH322QwaNIjBgwczaNAgevfu3azdLj3Gw9dZX/PBjg9YvGMx23K98/50SerC6J6jGd1zNBf0uIDUDql0aNEhrJKCJgClIl9YJgARcQI7gfHAQeBrYJox5tvq9u/fv7+59tpr2bJlC5mZmRw6dAiA1q1bM2rUKAYNGlT+c/bZZ9OlS5ewupj67Mnbw6d7PmXNgTWsObCGfSf2lb/XNrEtvdv2pmebnvRs05MebXrQo00PWsW3IsGZQLwzngRXQqXnTdm+cHTvbcQ54jj/vNVN9h1KqaYVrgngIuBpY8xV1uvfABhjnqthf5OYmMiQIUMYPHgwQ4YMYdy4cYwYMSKiR89mncpiQ/YGduftZlfeLr4/+T0HTh3g4KmD5BXl2RrbS0O9j09kJtkah1IqePn/nR+WCWAKcLUxZrr1+jbgAmPMA3773APcY70cAmwNeSCRKQU4ancQYULPRQU9FxX0XFQYaIwJetUo226vjTGzgFkAIpLRmCwWTfRcVNBzUUHPRQU9FxVEpFEjaJtq3oIsoKff6x7WNqWUUmGiqRLA18AAEekjIvHAzcDiJvoupZRSQWiSKiBjjFtEHgD+jbcb6BvGmOrXRvSa1RRxRCg9FxX0XFTQc1FBz0WFRp2LsBgIppRSqvlFxtzFSimlQk4TgFJKxSjbE4CIXC0iO0Rkt4jMsDuepiYib4hIjohs9dvWXkSWisgu67GdtV1E5G/WudksIiPsizz0RKSniCwXkW9FZJuI/MraHnPnQ0QSReQrEfnGOhe/s7b3EZF11u+cbnWqQEQSrNe7rfd72xl/qImIU0Q2isiH1uuYPA8AIrJPRLaIyCZft89Q/Y3YmgCsKSNmAtcAg4FpIjLYzpiawVvA1QHbZgCfG2MGAJ9br8F7XgZYP/cA/2imGJuLG3jEGDMYuBC43/r3j8XzUQxcbowZCgwDrhaRC4HngZeMMf2B48DPrP1/Bhy3tr9k7RdNfgVk+r2O1fPgc5kxZpjf+IfQ/I0YY2z7AS4C/u33+jfAb+yMqZl+797AVr/XO4Cu1vOuwA7r+at451Cqsl80/gAf4J0/KqbPB9AS2ABcgHfEq8vaXv73greH3UXWc5e1n9gde4h+/x7WRe1y4ENAYvE8+J2PfUBKwLaQ/I3YXQXUHTjg9/qgtS3WdDbGZFvPDwOdrecxc36sovtwYB0xej6sao9NQA6wFNgDnDDGuK1d/H/f8nNhvX8S6NC8ETeZvwCPAh7rdQdi8zz4GOBTEVlvTaEDIfobidyZ1qKUMcaISEz1zRWRJGAh8JAx5pT/TK+xdD6MMWXAMBFpC7wHnF3HR6KOiPwIyDHGrBeRS+2OJ0yMMcZkiUgnYKmIbPd/szF/I3aXAHTKCK8jItIVwHr0rVkZ9edHROLwXvznGGMWWZtj9nwAGGNOAMvxVnW0FRHfjZr/71t+Lqz3k4FjzRxqUxgN/FhE9gHz8FYD/ZXYOw/ljDFZ1mMO3huDUYTob8TuBKBTRngtBu6wnt+Bty7ct/12q2X/QuCkX7Ev4on3Vv91INMY82e/t2LufIhIR+vOHxFpgbctJBNvIphi7RZ4LnznaAqwzFiVvpHMGPMbY0wPY0xvvNeDZcaYW4ix8+AjIq1EpLXvOfBDvDMnh+ZvJAwaOCbgXTxmD/Bbu+Npht83DcgGSvHWz/0Mb53l58Au4DOgvbWv4O0ltQfYAoy0O/4Qn4sxeOs3NwObrJ8JsXg+gB8AG61zsRV40treF/gK2A3MBxKs7YnW693W+33t/h2a4JxcCnwYy+fB+r2/sX62+a6Rofob0akglFIqRtldBaSUUsommgCUUipGaQJQSqkYpQlAKaVilCYApZSKUZoAlFIqRmkCUEqpGPX/A2uWrQh0mb+ZAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aepGRk-CfgMX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "428416c0-6779-4841-ea1c-35c2b1bd5640"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d1P8BhBNgLKX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "635bad1d-f38a-4606-f70a-4f902bd0dc06"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "EPISODES = 500\n",
        "#np.save('/content/drive/My Drive/db/dqn/rewards',rewards)\n",
        "#np.save('/content/drive/My Drive/db/dqn/TRAIN_END',TRAIN_END)\n",
        "rw=np.load('/content/drive/My Drive/db/dqn/rewards.npy')\n",
        "te=np.load('/content/drive/My Drive/db/dqn/TRAIN_END.npy')\n",
        "\n",
        "rolling_average = np.convolve(rw, np.ones(100)/100)\n",
        "\n",
        "plt.plot(rw)\n",
        "plt.plot(rolling_average, color='black')\n",
        "plt.axhline(y=195, color='r', linestyle='-') #Solved Line\n",
        "\n",
        "plt.axvline(x=te, color='y', linestyle='-')\n",
        "plt.xlim( (0,EPISODES) )\n",
        "plt.ylim( (0,220) )\n",
        "plt.show()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xUZb748c93ZlKA0AlFioAUQVTQgAUUG1eUdREFhLXg2ta17OL6u8reva7rqlfXa1m94iq7qHiFEJrKFVEQKYKKhg6GLgihJHRCQpKZeX5/zJlkMqlkJjlTvu/XK6+ZeebMmScH8nzP08UYg1JKqfjjsDsDSiml7KEBQCml4pQGAKWUilMaAJRSKk5pAFBKqTjlsjsDAK1atTKdO3e2OxtxJT9/CwANG/a0OSdKqdpatWrVIWNMam0/HxEBoHPnzmRmZtqdjbiyZs1VAPTrt8TWfCilak9EdofyeW0CUkqpOKUBQCml4pQGAKWUilMaAJRSKk5pAFBKqTilAUAppeKUBgCllIpTETEPQNWv2av2kng8H4DFC7bYnBt7nDpxlDVLPiPv6GG7s6KUbTQAxJnTxR4en7mOCQMKAPifH7bbnKP6VXx0H/lbvuH4yjl4C05YqWJrnpSqndD3ctEAEGdOnnYD0LllI9o0SeanW4bZnKP68+c//5nnJj2HMYbBgwfz2muv0bdvX0Q0AKjos2jRIq677rqQzqF9AHHmVKEvADgd8VXozZo1i2effZY77riD3bt3s2TJEvr166eFv4pagwcPDvkcWgOIM3lxGADee+89Hn74YS655BImT55MQkKC3VlSKmQuV+jFt9YA4kxJAIiTO98dO3bw4IMPcuGFFzJ79mwt/JUKoDWAOBNPTUD5+fmMGDGC5ORkZs+ezVlnnWV3lpSKKBoA4ky8NAEZY3jiiSfYsGEDn3/+uRb+SlWg2iYgEekoIotF5EcR2SQiv7fSW4jIQhHZZj02t9JFRN4Qke0isl5ELqrrX0LVXDwEAGMMf/rTn5g4cSLjx4/n+uuvtztLSkWkmvQBuIHHjTG9gUuBh0WkNzABWGSM6Q4ssl4D3AB0t34eAP4R9lyrWvM3ATliOAD84Q9/4IUXXuD+++/nlVdesTs7SkWsagOAMWa/MWa19fwkkAW0B4YDU6zDpgA3W8+HAx8Yn++AZiLSLuw5V7WSV+hBJHY7gXNycpg4cSJ3330377zzDg6HjnNQqjJn9NchIp2BfsBKoI0xZr/11gGgjfW8PbAn4GN7rbTgcz0gIpkikpmbm3uG2Va1lXfaTaPE2O36mThxIsXFxTzxxBM6xl+patQ4AIhICjAbGG+MORH4njHGcIbzko0xk4wxacaYtNTUWu9prM7QqUI3KUmxGQA2b97MK6+8wsiRI+nVq5fd2VEq4tUoAIhIAr7Cf6oxZo6VfNDftGM95ljp2UDHgI93sNJUBMgrctMoyWl3NsLOGMMdd9xBw4YNefnll+3OjlJRoSajgASYDGQZY14NeGsuMM56Pg74JCD9Lms00KXA8YCmImWz00UeGiTGXgBYsWIFq1at4tlnn+Xss8+2OztKRQXxtd5UcYDIIOBrYAPgtZL/A18/wAygE7AbGG2MOWIFjDeBoUA+8GtjTGZV35HWuLHJvPjiUH4PVUNZ+0/gMYbi//gJgH7v97U5R+Gx6ccfOXrkCJdddhlOZ+wFOKUqIkuXrjLGpNX289U2BhtjllP5ernXVnC8AR6ubYZU3fIakBhb/riwsJBDubm079BBC3+lzkBk9Ab27AlLltidi7jw1FsraJTk4g8XPulLiIHr/vyf/8xzK1eyfelS6NrV7uwoVX9CHOmmg6TjQPaxAp7+ZCNFbi/FHkOCM3b+2QsLC3nnnXcYNmwYXbXwV+qMREYNQNWpV77Ywpw12Qzo0pJij5fEGAoAM2bMICcnh0ceecTurCgVdWKnJFCVapmSCMCuw6cocntJcMXGP7vX6+Vvf/sbvXv3ZsiQIXZnR6moozWAOJDaOAmA3YdPUeTxkuCMjU7guXPnsmnTJj788ENd8kGpWtC/mjhQ7PEN9d19OD9mmoByc3N58sknOeecc7jtttvszo5SUUlrAHGgsNgDwPGC4pjpBH7mmWfYtWsXCxcuDMvWeErFo+gvCVS1Ct2++Xuniz0Uu70kRnkfQH5+Ph9++CGjRo3iyiuvtDs7SkWt6C4JVI2UBgAvhR5v1NcAMjIyOH78OPfff7/dWVEqqkV3SaBqpNDtawI67fZYfQDR3Qn81ltv0bt3b737VypEGgDiQGGxrwZwqtCNMUR1DeCHH34gMzOThx56SNf7VypE0VsSqBrzNwH5RwNF8zyAt956i5SUFO688067s6JU1IvekkDVmD8A+EVrDeDw4cNMnz6dO++8kyZNmtidHaWiXnSWBOqM+PsA/KJ1FNB7773H6dOn+e1vf2t3VpSKCdFZEqgzElwDiMZO4IKCAt544w2uuOIKzj//fLuzo1RM0Bk0cSAWmoDefvtt9uzZw5QpU+zOilIxoyZbQr4rIjkisjEgLUNE1lo/u0RkrZXeWUQKAt57uy4zryrn9ngZ8upSvvzxYMlMYL9oCwDGGN566y0GDhzI1VdfbXd2lIoZNakBvI9vi8cP/AnGmJLFV0TkFeB4wPE7jDGxsc9gFDtyqohtOXlMmLOBJskuUpJc5BW6ASsAeKs5QQSZN28e27dv5+mnn7Y7K0rFlGpvBY0xy4AjFb1n7f87GkgPc75UqEqa+Q2Fbi9NGySUvJUURZ3AxhieeeYZunTpoou+KRVmoZYEVwAHjTHbAtK6iMgaEVkqIldU9kEReUBEMkUkMzc3N8RsqGD+fX+N8Y0CCgwA0dQENH/+fDIzM/nP//xPEhISqv+AUqrGQi0JxlL27n8/0MkY0w/4AzBNRCocsG2MmWSMSTPGpKWmpoaYDRXMP0nW4JsJ3KRBaWtfNO0H8Prrr9OhQwed+KVUHah1ABARF3ALkOFPM8YUGmMOW89XATuAHqFmUtWe15RvAoqWmcAbNmxgwYIF3HfffXr3r1QdCKUkuA7YbIzZ608QkVQRcVrPuwLdgZ2hZVHVhtf4ln3weg1FnrIBIFo2hPntb39LamoqDz/8sN1ZUSom1WQYaDrwLdBTRPaKyL3WW2Mo3/l7JbDeGhY6C3jQGFNhB7KqW1b5XzIHINr6AL7//ntWrFjBU089RatWrezOjlIxqdphoMaYsZWk311B2mxgdujZUqHy1wDcXt9j4+TSAJCc4CDfllzV3MSJE0lJSWHcuHF2Z0WpmKUzgWOUVe7jsZ40SnLx0sgLOHnaTacWDTnys42Zq8ahQ4fIyMjgnnvu0UXflKpDGgBilNcfASxOgdFpHW3KzZmZPHkyhYWF2vavVB2L/MZgVSumbPmP0xEdQz+9Xi/vvPMOgwcP5rzzzrM7O0rFNA0AMcobFAEcURIAvv76a3766Sfuu+8+u7OiVMzTABCjggOAM0q2T/zggw9ISUlhxIgRdmdFqZinASBGeaOwCSg/P5+ZM2cycuRIGjVqZHd2lIp5GgBilAmuAURBAPjoo484efKkDv1Uqp5oAIhR0VgDmDJlCmeffTZXXnml3VlRKi5oAIhR5TqBI7wPYO/evXz55ZfcddddOBz631Kp+qB/aTGqXCdwhNcApk6dijGGu+66y+6sKBU3NADEqGiaB2CMYcqUKQwcOJBu3brZnR2l4oYGgBhS6PaUdP56ys0EjtwAkJmZSVZWlt79K1XPNADEiLxCNz3/83NeX+TbnC2amoCmTJlCUlISo0ePtjsrSsUVDQAx4pS14fu0lb5V3oJHAUXqTODCwkLS09O5+eabadasmd3ZUSquaACIEf41/v3r/5ebBxChTUCfffYZR44c0bH/StmgJhvCvCsiOSKyMSDtLyKSLSJrrZ8bA977o4hsF5EtInJ9XWVcleUv8AvdHiB65gFMmTKFtm3bMmTIELuzolTcqUkN4H1gaAXprxlj+lo/nwGISG98O4WdZ33mLf8WkapueYN2AIuGPoDc3FzmzZvHHXfcgculK5MrVd+qDQDGmGVATbd1HA5MtzaH/wnYDgwIIX+qhgy+At9f7pcPAPWdo+qlp6fjdru1+Ucpm4RSLDwiIuutJqLmVlp7YE/AMXutNFXHgsf9B7+OxJnAU6ZMoV+/fvTp08furCgVl2obAP4BnAP0BfYDr5zpCUTkARHJFJHM3NzcWmZD+QUX+JHeBLRx40ZWr16td/9K2ahWAcAYc9AY4zHGeIF/UtrMkw0E7jvYwUqr6ByTjDFpxpi01NTU2mRDBQgs8I/nF0d8J/AHH3yAy+XiV7/6ld1ZUSpu1SoAiEi7gJcjAP8IobnAGBFJEpEuQHfg+9CyqGoisLy/8K8LIroG4Ha7+fDDD7nxxhvR4K+UfaodeiEi6cBVQCsR2Qs8DVwlIn3xlTu7gN8AGGM2icgM4EfADTxsjPHUTdZVoOBN4CN5HsBXX33F/v37dekHpWxWbQAwxoytIHlyFcc/DzwfSqZU6Lzesq8jaSbw1KlTadq0KcOGDbM7K0rFtQgcHKhqI7jJJ1L3BM7Pz2fOnDmMHDmS5ORku7OjVFzTABAjyo8CKvs6UvoAPv30U/Ly8rj99tvtzopScU8DQJTJOXGa3JOF5dKD7/gjdU/gqVOnctZZZ+m2j0pFAA0AUWbAfy2i//NflksPuuGPyBrAkSNHmD9/PmPHjsXp1BVClLKbBoAYEXzHH4l7As+aNYvi4mJt/lEqQmgAiBHRMBN46tSpnHvuufTt29furCil0AAQM4KbfCJtT+Cff/6ZZcuWcfvttyMRUBtRStVgHoCKDv7VQJskuzBEXg1g+vTpALr0g1IRRGsAMcI/8atpwwQ8XlO+E9jmu+6pU6dy6aWX0rVrV1vzoZQqpQEgRvjv+BOcDtweU74T2MZ/6Y0bN7J+/Xrt/FUqwmgAiDGJTgdur7fcqCCXjRFg2rRpOJ1ORo8ebVselFLlaQCIEYE1AK+BZVsPlXnfri4Ar9fLtGnTGDJkCK1bt7YnE0qpCmkAiAErth8ir9ANgMvpK+nnbdhf8r5DsG3kzTfffMPu3bu1+UepCKQBIMqdOF3MHZNX8vEa3747CRG2+e+0adNo0KABN998s91ZUUoFiazSQp2xIrcXY+BUkW/bhcQICgDFxcXMmDGD4cOHk5KSYnd2lFJBIqe0ULXib/svdvvGgSY4I2eS1RdffMHhw4e1+UepCFVtABCRd0UkR0Q2BqT9t4hsFpH1IvKRiDSz0juLSIGIrLV+3q7LzKvSGb9ua+C/q4IagF3t/9OmTaNly5Zcf/31tny/UqpqNakBvA8MDUpbCPQxxlwAbAX+GPDeDmNMX+vnwfBkU1WmpAbg8dUAIqUJKC8vj08++YRRo0aRkJBgd3aUUhWotrQwxiwDjgSlLTDGuK2X3wEd6iBvqgb8M379AaCiJqDgOQH14eOPPyY/P1+bf5SKYOG4XbwHmB/wuouIrBGRpSJyRWUfEpEHRCRTRDJzc3PDkI345N8M3u0pnQcQCaZNm8bZZ5/N5ZdfbndWlFKVCKm0EJE/AW5gqpW0H+hkjOkH/AGYJiJNKvqsMWaSMSbNGJOWmpoaSjbimv/mvriKPoD6lpOTw4IFCxg7diwOO9egUEpVqdZ/nSJyN/AL4HZjtTEYYwqNMYet56uAHUCPMORTVSJ4FFBiBU1A9d0JPHPmTDwej678qVSEq1UAEJGhwBPAL40x+QHpqSLitJ53BboDO8ORUVUxfwBwe/19AOX/ST3BS4PWsfT0dPr06cP5559fr9+rlDozNRkGmg58C/QUkb0ici/wJtAYWBg03PNKYL2IrAVmAQ8aY45UeGIVFqWdwFYfgMveJpfdu3ezYsUKxo4da2s+lFLVq3ZDGGNMRX/Jkys5djYwO9RMqZozQcNA7e4E9m/8MmbMGFvzoZSqnvbQRblyw0Bt3vkrPT1dN35RKkpoAIhyJX0AEdAElJWVxbp167T5R6kooQEgygXPBLazCSg9PR2Hw6EbvygVJXRT+ChljEFEyq0FFDwM9P4rulBQ7KmX/KSnp3P11VfTtm3bOv8+pVToNABEKY/X4HJKuRpA8ESwPw3rXS/5WbVqFdu3b2fChAn18n1KqdBpE1CU8nf+lhsGalMT0LRp00hMTOSWW26x5fuVUmdOA0CU8t/5e4MWerNjPwCPx0NGRgY33HADzZs3r/fvV0rVjgaAKOWf3Ru80qcdy0F//fXX7Nu3T0f/KBVlNABEKU9JDaBsuh2LwaWnp9OoUSNuuummev9upVTtaQCIUsbX51uyHLRffTcBFRUVMWvWLIYPH07Dhg3r9buVUqHRABClKqsB1HcT0IIFCzhy5Iiu/KlUFNIAEGE8XsOzn/7I/uMF1R4HFez2Vc99wOnp6bRo0YIhQ4bU7xcrpUKmASDC/LDrCJOX/8S/z1xf5XHeSmoA1OPKz/n5+XzyySeMHDmSxMTE+vtipVRYaACIMMHr+1d3XPAw0LZNkxmdVj9bNP/f//0fp06d0tE/SkUpDQCRpoZ38P4moOAA4HQIL428MNy5qlB6ejpnnXUWV1xR6dbPSqkIpgEgQkkFjfmB7f3+CkJwF4CjnrZ/PHr0KJ999hm33XYbTqezXr5TKRVeNQoAIvKuiOSIyMaAtBYislBEtlmPza10EZE3RGS7iKwXkYvqKvPxJrC9v7ImoPoyZ84ciouLdfSPUlGspjWA94GhQWkTgEXGmO7AIus1wA349gLuDjwA/CP0bMaPqorzwMK+smGgjnraECY9PZ1u3bpx8cUX18v3KaXCr0YBwBizDAje23c4MMV6PgW4OSD9A+PzHdBMRNqFI7PxpKKWnMCbfW8lfQD1UfwfOHCAxYsXM3bsWKSempyUUuEXSh9AG2PMfuv5AaCN9bw9sCfguL1WWhki8oCIZIpIZm5ubgjZiB8V1QCC5wHURx/AjBkz8Hq9OvpHqSgXlk5g4yuFzqgx2hgzyRiTZoxJS01NDUc2YkJVTfplawDWY/A8sHq4IZ82bRoXXnghvXr1qvsvU0rVmVACwEF/0471mGOlZwMdA47rYKWpM1BhE1BAjPWW1ACq/1w47dy5k5UrV2rnr1IxIJQAMBcYZz0fB3wSkH6XNRroUuB4QFORCkHg3X5l8wAqGj4aTtOnTwdgzJgxdfo9Sqm6V6MtIUUkHbgKaCUie4GngReBGSJyL7Ab8O8E/hlwI7AdyAd+HeY8xzRTRUtaYGFf2TDQuh4ElJ6ezsCBA+nUqVPdfpFSqs7VKAAYYyrr7bu2gmMN8HAomYpn/vK84olgpc8rbwKquwiwYcMGNm7cyJtvvlln36GUqj86EzjCeKroBQ4c8eMp6QSuvxpAeno6TqeTUaNG1d2XKKXqjQaACBO8wUuZ9yrsAyh7TF31ARhjmD59Otdddx2tW7euk+9QStWvGjUBqfrjL9grnghmyj0v1wlshfRp913CabcnbPlauXIlP/30E08//XTYzqmUspcGgAhTWdu+773S55VNBPPHjcu7tQprvtLT00lKSmLEiBFhPa9Syj7aBBRhgpt0HstYy1Mf+9bgK9sHUMlaQHXQCezxeMjIyGDYsGE0adIk7OdXStlDA0CECW4C+mhNNv/73W6g7FRrfywo1wRUB10Aixcv5uDBg7r0g1IxRgNAhKlqeWevTTWA9PR0GjduzLBhw8J+bqWUfTQARBhPTUcBVdIHEG6FhYXMnj2bESNG0KBBgzr9LqVU/dIAEGGqCgBldwQzZR79wl0D+Pzzzzl+/Lg2/ygVgzQARJiqmoDKzgQu++gX7olg6enptGrVimuvLTfpWykV5TQARBj/DN+KVLwjWHAncPgiQF5eHnPnzmXUqFEkJCSE7bxKqcigASDC+Av2igry4B3BPlmbzXPzssocE84awNy5cykoKNDmH6VilE4EizD+Nv1lW3P59Xvfl30vaBTQpGU7y30+nDWA6dOn0759ewYOHBi2cyqlIofWACJMYCfw4i1lt8r0lukDMOXG/Iez//fYsWN8/vnnjB49GodD/5soFYv0LzvCVNUJTAU7ggUKZ//vxx9/THFxMbfddlsYz6qUiiQaACJMjecBeMsP+QznENCMjAw6d+7MgAEDwnZOpVRkqXUAEJGeIrI24OeEiIwXkb+ISHZA+o3hzHCsq2o/gOBRQHXVBHT48GG+/PJLbrvttjrdYEYpZa9adwIbY7YAfQFExIlv4/eP8G0B+Zox5uWw5DDOVLUfQGBsKCz2VBAAwlNYz5kzB7fbrc0/SsW4cDUBXQvsMMbsDtP54lZN5wHkF3nKbf4Srnv1jIwMunfvTt++fcN0RqVUJApXABgDpAe8fkRE1ovIuyLSvKIPiMgDIpIpIpm5ubkVHRKXqt4SsvR5QbGn3Jj/cPQBHDx4kMWLF2vzj1JxIOQAICKJwC+BmVbSP4Bz8DUP7QdeqehzxphJxpg0Y0xaampqqNmIGZU1ARljygaAIk+5Rv9wlNezZs3C6/Vq849ScSAcNYAbgNXGmIMAxpiDxhiPMcYL/BPQYSRnoLIagNtrgpqA3OWafMJRA8jIyKB379706dMn5HMppSJbOALAWAKaf0SkXcB7I4CNYfiOuFFZDSC/yFOuDyC4CSjU4j87O5vly5fr3b9ScSKkACAijYAhwJyA5JdEZIOIrAeuBh4L5TviTWXzAC58ZgF5he6S1wVFnnJt9KFWAGbOnIkxRgOAUnEipLWAjDGngJZBaXeGlKM4V8UoUE4UlAYA3yigskLttM3IyKBv37707NkzpPMopaKDzgSOMFUtBeFv8nGIbxRQcHkfykqgu3fv5rvvvtO7f6XiiAaACFPVUhDF1nuNEl2VNAHVPgLMmDEDgNGjR9f6HEqp6KIBIMJUNQ/Abc0Sa5TkIr+4olFAtf/ejIwM+vfvT9euXWt/EqVUVNEAEGGqWgqiuCQAOK0aQPARtYsA27dvZ9WqVdr8o1Sc0QAQYapsAvJYTUBJLmsYaPBqoLX7zoyMDECbf5SKNxoAIkxVTUD+GkDDRCcFxZ5y79e2CyAjI4PLL7+cjh071u4ESqmopAEgwlTVBOS2agApSS6MKQ0IfrWZCZyVlcWGDRu0+UepOKQBIAIUub3sP14AgKeKeQDFXn8NwDd9I7gWUJsKQEZGBiLCqFGjavFppVQ00wAQAZ6cvZ7LXviKQren6k5gd2kfAFgLwgWozTDQjIwMBg8eTLt27ao/WCkVUzQARIAFmw4AvppAVZ3AbqsGkJLkBOB0cdkmoDMt/0+dOsXmzZu1+UepOKUBIAL4i3yvt7pOYN97/iagU0WlS0M45Mz7AHJycnA6ndx6661nlmGlVEzQABBBir3eGs8DAMg7XRoAnA454xpATk4O11xzDbofg1LxSQNABHF7TI2Ggfr7ANwBwcLpkDOqAZw8eZLTp09r849ScUwDQATwl/nFnqr7AEomgiWWX8TV5XCc0Sig3NwcRGDEiBFnklWlVAzRABBBgnf9Kvd+UA0g0Jk0ARljyMnJpUWLFrRo0aJWeVVKRb9w7Am8y9oAZq2IZFppLURkoYhssx4r3BheleWutgZgBYBEZ7n3XA6p8TDQ7777jsLCQlJTW9cuo0qpmBCuGsDVxpi+xpg06/UEYJExpjuwyHqtKmGscUDFHoPXW/lxJctBV1IDqOlaQBkZGTgcQqtWrc44r0qp2FFXTUDDgSnW8ynAzXX0PTHB3+rj9nprtBx0g8pqADXoBfB6vcycOZMWLVrgdJY/j1IqfoQjABhggYisEpEHrLQ2xpj91vMDQJvgD4nIAyKSKSKZubm5YchG9DteUMyq3UfLpb9224VA6VpAToeQnFD2n85Rwz6A5cuXs2/fPlq31uYfpeJdSHsCWwYZY7JFpDWwUEQ2B75pjDEiUu621hgzCZgEkJaWVsUKOPHjzsnfV5jetEECAEVWDcAhQsNEF6eLi0qOqWkfwPTp02nQoAEtWrSs9lilVGwLuQZgjMm2HnOAj4ABwEERaQdgPeaE+j2xrLrol+TyNdX4O4EFaJBQtvmmJn0ABQUFpKenc/PNN2vzj1IqtAAgIo1EpLH/OfBvwEZgLjDOOmwc8Eko3xNPXh/Tl7ZNksukJbp8/0z+JiBfDaBsAe5yOKqdCDZnzhyOHTvGfffdF8YcK6WiVag1gDbAchFZB3wPzDPGfA68CAwRkW3AddZrVQMVTfJKdPr+mUpqAEK5AFCTeQD/+te/6Nq1K1dddVVY8qqUim4h9QEYY3YCF1aQfhi4NpRzx6tEl6PcZLCkBH8A8KWLQHJQE5DLWXUfwNatW1myZAnPP/88DofO/1NK6UzgyBBQ3ie5HATPBfPXAPzLQQc3AXVrneKrAVTxFZMmTcLlcnHPPfeEK9dKqSinASDCJLocmKAaQIIVALYezANKRwEB3H5JJ778w2BcVXQCnz59mvfee4+bb76Ztm3b1l3mlVJRRQNAhElyOXno6m4ANE72FfLBLTsipZPBnFap75DKm4Bmz57NkSNH+M1vflNHuVZKRSMNABEm0eXg3kFd2PXiMJ64vicAzRsmljkmsBPYP/Kn91lNOLdt4wrPOWnSJLp168Y111xThzlXSkWbcEwEUyEyAZ0ASa7SmHznZZ2587LO5ZqEHCLlagBP33RehefevHkzy5Yt429/+5t2/iqlytASIcIEBgC/4KadwIlgrmpmf7399tskJCRw9913hyuLSqkYoQEgwvhn/VanpAmoigCQlZXFW2+9xa9+9Std+0cpVY4GgAgQ2MKTlFD9P0mjJBcNrFFAzko6fo0xPPjgg6SkpPDSSy+FJZ9KqdiifQARxj/mvzILH7uS5AQnDROqrgEsX76cZcuWMXHiRL37V0pVSGsAESBwD4CqmnSgdAawvwmooj4Ar9fLCy+8QNOmTRk3bly595VSCrQGYDuP11DFHjDl+ANA8CigQBMnTmT+/Pm89tprNGrUKCz5VErFHq0B1BGP1/DPZTvJL3JXeZx/gbea8hf8/lFAwSuAHjhwgKeffprrrruO3//+92d0bqVUfNEAUEfmb9zP859l8SbndY0AABDSSURBVPIXW8uke72Gt5fu4HhBMXDmASDZGibqXwoiuMvg0UcfpaCggDfeeKPGm8QrpeKTBoA6kl/kAeBYQVGZ9B92HeHF+ZtJe24hK3ceLlnjv6ZcVolf2gRU+k+4du1aZs2axZNPPkmvXr1Cyb5SKg5ERADYczSfIveZ3QlHupJ776Dy3d/JW+wx3DbpO4q9tfu9/Z3AzoCb/L/+9a80bdqU8ePH1+qcSqn4EhEB4Fh+MYfyCu3ORlhV1vzirxn4nWkNwK9VShJX9kilb6fmgO/u/6OPPuKxxx6jWbNmtTqnUiq+1DoAiEhHEVksIj+KyCYR+b2V/hcRyRaRtdbPjTU5X87J2AoAbqttP7h4zy8s2yl84MTpWp0/0eXgg3sG0LdjMzZs2MCtt95K06ZNteNXKVVjoQwDdQOPG2NWW/sCrxKRhdZ7rxljXj6Tk+XUsiCMVP47/eCF3PKCAsD6PcdC+p6jR48ydOhQvF4v06ZN07t/pVSN1boGYIzZb4xZbT0/CWQB7Wt7vtwYawIqKLYCQEDalgMn+fuX2wD46KHLfWkHT9b6O44ePcrdd9/N/v37+fTTT7nxxhpVtpRSCghTH4CIdAb6ASutpEdEZL2IvCsizSv5zAMikikimQA5J2IrAJyy7vQDh3ne9OZyso8VAHBWswYAZB/z1XwGdG7Bf9x4bo3ObYxh1qxZDBo0iHnz5vGXv/yFiy++OJzZV0rFgZBnAotICjAbGG+MOSEi/wCexXfz+yzwClBuI1pjzCRgEkCj9j1MrPUB+JuAThWWdvoGjnRq2ci3ycs+KyA8P6IP3dtUvKELwONDepDgcpCXl8ftt9/O3Llz6dSpE3PnztU7f6VUrYQUAEQkAV/hP9UYMwfAGHMw4P1/Ap9WmwmHg9yTpX0AHq/hpc83M7p/R85JTQkli7bxzwD21wSC+wJcTgcpSa6SAJCSXPU/xaPXdgdg+PDhzJs3j5dffpnx48fjdNZs+WillAoWyiggASYDWcaYVwPS2wUcNgLYWN25XE7hUF7phKlN+47zzrKd/PvMdbXNnu1KagDW4ysLtpY7pnGyq+S4lKSqA8CxY8d49dVXmTt3Ls899xyPP/64Fv5KqZCE0gcwELgTuCZoyOdLIrJBRNYDVwOPVXcip0M4ll8aANZZI2NOF9f/5LCJi7ezcufhKo/5ZschXpy/ucxd/YnTxUyYvZ5cqynLX7AfzivkzskreXPx9nLnaZKcAPj2+G2UWHEAyM3N5ZFHHqFFixY8/vjjDBw4kN/97ne1+t2UUipQrZuAjDHLCZjwGuCzM86EQziaX1zyetXuowCcqmYhtXAzxvDfX2wBYNeLw8q89z+LtnF+h6Zc1bM1L87fzPq9x+ncsiFjBnQCYNLSnUz/YQ/tmzXg0Wu7lzQB5ZwsJOdkIT3apDDjN5fR968LS87ZpIHv8qckuipcBvrbb79l1KhRZGdnc//993PffffRv39/XeNHKRUWETET2OkQTpwuLpk8tetwPgA/H8mnIGjmbF2qrMZR5PbyysKt3P3eD0DpGvwLfyzp7uD7XUcAOGYt8hY84/eVUX1p2iChTJq/BtC4gvb/v//971x++eUUFxfz/fffM2nSJAYMGKCFv1IqbCIiALgcgjGUrJB5+JSvGcWY2s+UrY3AhdsCh2/+dOhUyfMnZ63n5yO+jtutOb4x/MYYNmUfB2Dy8p/4dP2+chO+WqYkliu8m1gBIbADOD8/n1GjRvHYY49xyy23sH37dvr37x+OX08ppcqIiADgX9HS3wx0OK+I7q1TrOf1NzzUH4CgbKEfOFkrI3MPh/IKaZDgZM+RAm76n+VMXflzSWcvwCPT1rDr0Ck6tmhQktbCGvb59E29efuOi4DSO/+UJBfGGL766ituvfVWZs2axVNPPcWMGTNo3LjyoaFKKRWKiNgRzOUQ3MCx/CIKijzkF3no0aYx23LyyowOqmvHA/ohZmbu4cf9J0ry43QIk8ellTQDXdOrNfPW72dD9nH2H/fVCH5zZVemfLuL08VevAbuv6Irf/5kE1C6k9evB3Yp+Q5/E5D78B6GDRvG/Pnzady4MS+//DKPP/54ffzKSqk4FhEBwL+t4dH84pLmnx5tGjNvw36OnAotADz18UaGXdCOS7u2rPZYf/u9CPxr+U8Y49t5q6DYQ9fURvTv3KLk2Meu60GDBCdZ+0+wad8JAEaldeDha7oxYfZ6WjdO5qoerYFNFX7XyZMnKdi0iIQV81m46isaNkjmhRdeYPz48SQnJ4f0OyulVE1ESBOQLwC8unArL1ujcLq38TUBfbPjEA9PXc3p4oo7gzfsPc79H2RWuPXisfwi/ve73Yz953c1yoe/Cej2SzphjG+27uAeqQD0bNOYRkku/t+/9eCfd6XRrXUKL4+6kMsCAkuH5g1pkpzAW7dfzF9+eR5tmiaVOb8xhnXr1vHoo4/Svn17/vz4wxzN+pZbbxnB2rVrmTBhghb+Sql6ExE1gASXgySXg6z9J8ja77ubbtc0mcZJLj5dvx+AMQM6ckV3X2H8r6938snafTw59FzumOxbfmj2qr3MXp3NpV1bMuGGc9l9+BSj3/kWoGTT9ZmZe9iek8cfb6x4tyx/E9Dtl5zNh9/9TL9OzTm7VUMAOrX0PT5yTfcyn/Gnt0pJLGnm8UtyOfEWnqJg11ruu+8jPv/8c7Kzs0lMTGTMmDE89NBDOrJHKWWbiAgAgm+DE/9CaQAtGyXRMiWRk9ZomvV7j3NRp+Y8NHU1S7fmAnD/B5klx//9y20cPlXE2j3H+GLTgTKduAD/9VkWk5btBOCeQV1o06T0Tjvjh59Zt/c4zRok4HQI57ZtzBNDe3Jxp+ZkWnMS/O31wVql+O7y/U1Mxhh27drF0qVLef/998levhyvx8Ospk0ZMmQIN9xwAzfddBOpqamhXDKllApZRAQAgLZNk8sEgPbNG9CiUWLJnIDXF23j3eU/cfhUEbdc1J4GCU6mrvyZJJcDY+BwQF9BcOEPlBT+ACMmrqBRkovmDRO594ouPDl7A+DbZrFJsgsR4aGrugHQ+6wmHMor5K7Lzq4w31f1SOWXXRz0OLmau+76O1999RXZ2dkAnHPOOUx48kluuOEGLr30UlyuiLncSikVOQHg1dEXkv79HrzGMPLiDjgdwr2DutJ+0wESnMKc1dkcdhfx0sgLGJ3WkT1H8skrdHNJl5bMXr2XVbuPMnZAJ1qlJNKheYOSQj3YPQO7cOCEL9As23qI3/zvKlqlJDG871nsP17AxWe3KHN84+QEnr7pvJLXbrebdevWsXz5cr7++muWL1/OwYO+CWGtWrXimmuuYfDgwQwcOJA+ffroej1KqYgVMQHg7JaNmHBD2fXwh13QjmEXtMPt8dKxeUOGXdCOHtaSyR1bNOT1Mf0AaNMkibnr9nHnpWfT+6wmGGM4lFdEotPBpK93knuykPbNGvDmr/rRr1Pp9gTLtx1i5qo9jLu8Mxd1qnDbAk6ePMmaNWv45ptvWLJkCStWrCAvLw+Azp07M2TIEAYNGsSgQYPo1asXDkdE9KsrpVS1JHiZYjukpaWZzMzM6g+shWP5Rfzz65387truJLmqvhv3er1s2bKFpUuX8u233/LDDz+weXPpom/nnXcegwcP5oorrmDQoEF06NChTvJcH9asuQqAfv2W2JoPpVTticgqY0xabT8fMTWAutKsYSL/fn3ZmsXx48fZsmULW7duLfO4bds28vN9fQ5t2rShf//+jBkzhrS0NPr3768dt0qpmBKzAaCoqIidO3eWK+S3bNlCTk5OyXEOh4MuXbrQs2dPrr76as4//3yuvPJKzjnnHB2eqZSKaVEfAI4dO0ZWVhZZWVn8+OOPbN68ma1bt7Jz5048ntLJY6mpqfTs2ZNf/OIX9OzZkx49etCzZ0/OOeccEhMTbfwNlFLKHlERAIwxZGdns3nz5pLC3v/8wIEDJcclJyfTo0cP+vbty+jRo0sK+h49etC8ecWdvEopFa/qLACIyFDgdcAJ/MsY82JNP3vw4EGWLFnC4sWLWb16NZs3b+bkydIVOZs2bUqvXr0YOnQovXr1onfv3vTq1YvOnTvrsEullKqhOgkAIuIEJgJDgL3ADyIy1xjzY0XHHzt2jMcee4wNGzaQlZXFvn37AGjcuDEDBgxg3Lhx9OrVi169enHuuefStm1bbZ9XSqkQ1VUNYACw3RizE0BEpgPDgQoDwI4dO3j77bfp06cP1113HX369GHw4MFcdNFFOntWKaXqSF2Vru2BPQGv9wKXBB4gIg8AD1gvC0+fPr0xMzOTupoPEEVaAYfq7+siuiZVz9cioum1KKXXolTPUD5s2+21MWYSMAlARDJDmcwQS/RalNJrUUqvRSm9FqVEJKQ75rpatyAb6BjwuoOVppRSKkLUVQD4AeguIl1EJBEYA8yto+9SSilVC3XSBGSMcYvII8AX+IaBvmuMqXhvRJ9JdZGPKKXXopRei1J6LUrptSgV0rWIiMXglFJK1T9du1gppeKUBgCllIpTtgcAERkqIltEZLuITLA7P3VNRN4VkRwR2RiQ1kJEForINuuxuZUuIvKGdW3Wi8hF9uU8/ESko4gsFpEfRWSTiPzeSo+76yEiySLyvYiss67FM1Z6FxFZaf3OGdagCkQkyXq93Xq/s535DzcRcYrIGhH51Hodl9cBQER2icgGEVnrH/YZrr8RWwNAwJIRNwC9gbEi0tvOPNWD94GhQWkTgEXGmO7AIus1+K5Ld+vnAeAf9ZTH+uIGHjfG9AYuBR62/v3j8XoUAtcYYy4E+gJDReRS4G/Aa8aYbsBR4F7r+HuBo1b6a9ZxseT3QFbA63i9Dn5XG2P6Bsx/CM/fiDHGth/gMuCLgNd/BP5oZ57q6ffuDGwMeL0FaGc9bwdssZ6/A4yt6LhY/AE+wbd+VFxfD6AhsBrf7PlDgMtKL/l7wTfC7jLrucs6TuzOe5h+/w5WoXYN8Cm+6epxdx0CrscuoFVQWlj+RuxuAqpoyYj2NuXFTm2MMfut5weANtbzuLk+VtW9H7CSOL0eVrPHWiAHWAjsAI4ZY9zWIYG/b8m1sN4/DrSs3xzXmb8DTwBe63VL4vM6+BlggYisspbQgTD9jehKaxHGGGNEJK7G5opICjAbGG+MORG40ms8XQ9jjAfoKyLNgI+Ac6v5SMwRkV8AOcaYVSJyld35iRCDjDHZItIaWCgimwPfDOVvxO4agC4Z4XNQRNoBWI/+PStj/vqISAK+wn+qMWaOlRy31wPAGHMMWIyvqaOZiPhv1AJ/35JrYb3fFDhcz1mtCwOBX4rILmA6vmag14m/61DCGJNtPebguzEYQJj+RuwOALpkhM9cYJz1fBy+tnB/+l1Wz/6lwPGAal/UE9+t/mQgyxjzasBbcXc9RCTVuvNHRBrg6wvJwhcIRlqHBV8L/zUaCXxlrEbfaGaM+aMxpoMxpjO+8uArY8ztxNl18BORRiLS2P8c+DdgI+H6G4mADo4bga342jv/ZHd+6uH3TQf2A8X42ufuxddmuQjYBnwJtLCOFXyjpHYAG4A0u/Mf5msxCF/75npgrfVzYzxeD+ACYI11LTYCf7bSuwLfA9uBmUCSlZ5svd5uvd/V7t+hDq7JVcCn8XwdrN97nfWzyV9GhutvRJeCUEqpOGV3E5BSSimbaABQSqk4pQFAKaXilAYApZSKUxoAlFIqTmkAUEqpOKUBQCml4tT/B17YNjvG6YXzAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0elKpgePUmB9",
        "colab_type": "text"
      },
      "source": [
        "**Changes**  \n",
        "*hyper parameters*: You can alter alpha, gamma, batch size, and episode length to see what differences the algorithm returns.  \n",
        "*Training End*: You can also change the line where I only check the last 5 runs before switching to testing mode (if len(rewards) > 5 and np.average(rewards[-5:]) > 195:) as that doesn't prove it was solved. The reason I did this was because I wanted to limit the amount of runs I made.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B19ILOAfUmB-",
        "colab_type": "text"
      },
      "source": [
        "**Conclusion**  \n",
        "This is a Deep Q-Network implementation. There are some changes you can make here and there but it follows the paper. Hopefully, you were able to understand the code as well as make your own version to compare with this one."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "AMyF_HBtUmB_",
        "colab_type": "text"
      },
      "source": [
        "**Reference**  \n",
        "Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Petersen, S. (2015). *Human-level control through deep reinforcement learning*. Nature, 518(7540), 529"
      ]
    }
  ]
}